1

I'm getting an error trying to run CockroachDB (v19.2.1) on AKS. I'm able to spin up cluster using the default storage class. However, when I attempt to specify AzureFile storage, I get errors.

Storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cockroach
  namespace: cockroach
provisioner: kubernetes.io/azure-file
mountOptions:
  - mfsymlinks
  - dir_mode=0777
  - file_mode=0777
  - uid=1000
  - gid=1000
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS

Helm command:

helm install cockroach stable/cockroachdb `
    --namespace cockroach `
    --set StorageClass="cockroach" `
    --set Storage="100Gi"

The first node starts up, but the second node crashes with the following:

F191203 17:07:35.042824 214 storage/store_raft.go:322  [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]

Full log:

CockroachDB node starting at 2019-12-03 17:07:34.142182444 +0000 UTC (took 6.1s)
build:               CCL v19.2.1 @ 2019/11/18 23:23:55 (go1.12.12)
webui:               http://0.0.0.0:8080
sql:                 postgresql://root@cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257?sslmode=disable
RPC client flags:    /cockroach/cockroach <client cmd> --host=cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 --insecure
logs:                /cockroach/cockroach-data/logs
temp dir:            /cockroach/cockroach-data/cockroach-temp261131495
external I/O path:   /cockroach/cockroach-data/extern
store[0]:            path=/cockroach/cockroach-data
status:              restarted pre-existing node
clusterID:           dd364f3f-7900-45f2-9a65-7a105cd7a147
nodeID:              3
I191203 17:07:34.153647 232 server/server_update.go:53  [n3] no need to upgrade, cluster already at the newest version
I191203 17:07:34.191164 234 sql/event_log.go:130  [n3] Event: "node_restart", target: 3, info: {Descriptor:{NodeID:3 Address:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 Attrs: Locality: ServerVersion:19.2 BuildTag:v19.2.1 StartedAt:1575392852575521409 LocalityAddress:[] ClusterName: SQLAddress:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257} ClusterID:dd364f3f-7900-45f2-9a65-7a105cd7a147 StartedAt:1575392852575521409 LastUp:1575392800598285198}
I191203 17:07:34.360523 214 storage/replica_raftstorage.go:794  [n3,s3,r10/2:{-}] applying LEARNER snapshot [id=ea03bb31 index=18]
I191203 17:07:34.487170 214 storage/engine/rocksdb.go:119  [db/version_set.cc:3086] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
I191203 17:07:34.508684 84 gossip/gossip.go:1531  [n3] node has connected to cluster via gossip
I191203 17:07:34.508951 84 storage/stores.go:259  [n3] wrote 2 node addresses to persistent storage
I191203 17:07:34.711691 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711725 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711873 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711893 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711990 214 storage/replica_raftstorage.go:815  [n3,s3,r10/2:{-}] applied LEARNER snapshot [total=351ms ingestion=4@-9223372036855ms id=ea03bb31 index=18]
E191203 17:07:35.042651 214 util/log/crash_reporting.go:537  [n3,s3,r10/2:{-}] Reported as error 2cfee4bb679a4beb8a489e30d87ab8ea
F191203 17:07:35.042824 214 storage/store_raft.go:322  [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:908
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:526
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
while applying snapshot
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:528
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
goroutine 214 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000438300, 0xc000438360, 0x0, 0xc0031ba280)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1017 +0xb1
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x7508900, 0xc000000004, 0x6cd15ce, 0x15, 0x142, 0xc00051ac00, 0x1087)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0x95b
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x4a54f40, 0xc003763560, 0xc000000004, 0x3, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x4a54f40, 0xc003763560, 0x2, 0xc000000004, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.FatalfDepth(...)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:199
github.com/cockroachdb/cockroach/pkg/storage.maybeFatalOnRaftReadyErr(0x4a54f40, 0xc003763560, 0x4104925, 0x17, 0x49e0600, 0xc003a98600, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:859 +0x1de
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x4a54f40, 0xc003763560, 0xc003694000, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:322 +0x25f
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000c48300, 0x4a54f40, 0xc003763560, 0xc00077dbc8, 0xc0035b5418, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200 +0x16b
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0xe5401ee131bb03ea, 0x3f9d1493005c399, 0xc000545340, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262 +0x1a4
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0x7f30769bb2c8, 0xc000369540, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854 +0x6d3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1(0x4a54f40, 0xc0035d5bf0, 0xc00054d320, 0x411c3aa)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59 +0x1d9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00054d320, 0x4a54f40, 0xc0035d5bf0, 0x411c3aa, 0x1e, 0xc00331bc30, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc000c48300, 0xc00077db80, 0x7f30769bb298, 0xc000369540, 0xc000369540, 0x30)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49 +0xe6
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x4aa3720, 0xc000369540, 0xc000ca40f0, 0x4a54f40, 0xc0035d5b30, 0x78c3df, 0xc000544ff0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x4a54f40, 0xc0035d5b30)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00054d320, 0x4a54f40, 0xc0035d5b30, 0xc000075780, 0x32, 0x0, 0x0, 0xc0035d5b60)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131

Do I need to specify different options in my storage class?

1

AzureFile didn't support that operation, so I tried AzureDisk and it worked. This is what I used:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cockroach-disk
  namespace: cockroach
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Standard_LRS
  kind: Managed
  #resourceGroup: CockroachStorage
reclaimPolicy: Retain
allowVolumeExpansion: true

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.