Nexus database restore
!!!Make sure that you always have a backup of the database you’re going to touch.!!!
To restore the Nexus database in case of corruption the steps below can be followed. There are two scenario’s, a repair and a drop. Both methods will be described below.
2022-04-07 07:00:51,728+0000 INFO [periodic-8-thread-1] *SYSTEM org.sonatype.nexus.rapture.internal.HealthCheckCacheManager - Health check status changed from true to false for Transactions
2022-04-07 07:01:07,477+0000 WARN [qtp1175408681-435] blaataap-jenkins org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 44347e4c-081d-474f-a1ce-976edf0f8955 (/nexus-data/blobs/default/content/vol-39/chap-24/44347e4c-081d-474f-a1ce-976edf0f8955.properties)
2022-04-07 07:01:07,479+0000 WARN [qtp1175408681-435] blaataap-jenkins org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@FB02D905-3A3BF312-6B38847A-5FD6C879-5478D5A4:44347e4c-081d-474f-a1ce-976edf0f8955 exists in metadata, but is missing from the blobstore)
2022-04-07 07:01:07,481+0000 WARN [qtp1175408681-435] blaataap-jenkins org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/maven-snapshots/com/blaat/things/sit/sit-parent/maven-metadata.xml
org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:228)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy222.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:83)
at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
at org.sonatype.nexus.repository.maven.internal.orient.MavenFacetImpl.get(MavenFacetImpl.java:229)
at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.doGet(HostedHandler.java:74)
at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.handle(HostedHandler.java:60)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:228)
at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy222.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:83)
at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
2022-04-07 07:16:01,870+0000 WARN [qtp1175408681-634] blaataap-jenkins org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/maven-public/com/blaat/things/sit/sit-base/maven-metadata.xml
java.io.IOException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
There are 3 databases:
component config security
Repair
Edit the Nexus deployment to stop Nexus
kubectl edit deployments.apps nexus
# set replicas to 0
Make a dump from the deployment and edit it so the pod can be started without Nexus itself. This needs to be done to reach console. See example below:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: nexus
name: nexus-fix
namespace: platform
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nexus
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nexus
spec:
containers:
- env:
- name: INSTALL4J_ADD_VM_PARAMS
value: -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=4096m
...
image: ghcr.io/blaataap/nexus:1.2.3
imagePullPolicy: IfNotPresent
command: ["sleep", "10000000"]
name: nexus
...
volumes:
- name: nexus-data
persistentVolumeClaim:
claimName: nexus-data
- name: nexus-blobs
persistentVolumeClaim:
claimName: nexus-blobs
Deploy and start debug pod
kubectl apply -f nexus-fix.yml
kubectl exec -it nexus-fix-74b7dc5f99-j4h9m -- bash
Sometimes there are empty wal files, remove them
cd nexus-data/db/config/
ls -latrh *wal
rm config.1.wal
Repair config database
Start console (in the root or homedir, the history file needs to a rw filesystem
cd
java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar
connect plocal:/nexus-data/db/config admin admin
rebuild index *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
REPAIR DATABASE component
DISCONNECT
Repair component database
Start console (in the root or homedir, the history file needs to a rw filesystem
cd
java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar
connect plocal:/nexus-data/db/componentadmin admin
rebuild index *
When corruption is found during index rebuild it’s shown as follows:
Rebuilding index(es)...$ANSI{green {db=component}} Error during index rebuild
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #28:129572: found duplicated key 'OCompositeKey{keys=[#18:0, null, com/blaat/blaataap/thingsengine/api-actuators/maven-metadata.xml.sha256]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #28:129570
The record can be deleted as follows
load record #28:129570
truncate record #28:129570
rebuild index asset_bucket_component_name_idx
#rinse and repeat till no errors are found
REPAIR DATABASE component
DISCONNECT
exit
When rebuild/re-index is done stop de debug pod and start Nexus again
kubectl delete -f nexus-fix.yml
kubectl edit deployments.apps nexus
#set replicas to 1
Source:
https://help.sonatype.com/repomanager3/planning-your-implementation/backup-and-restore/restore-exported-databases
https://mariosk.wordpress.com/2019/09/07/how-a-nexus-repository-manager-corruption-led-to-a-mini-odyssey/
https://gist.github.com/marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f