I'm getting an error trying to run CockroachDB (v19.2.1) on AKS. I'm able to spin up cluster using the default storage class. However, when I attempt to specify AzureFile storage, I get errors.
Storage class:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cockroach
namespace: cockroach
provisioner: kubernetes.io/azure-file
mountOptions:
- mfsymlinks
- dir_mode=0777
- file_mode=0777
- uid=1000
- gid=1000
- nobrl
- cache=none
parameters:
skuName: Standard_LRS
Helm command:
helm install cockroach stable/cockroachdb `
--namespace cockroach `
--set StorageClass="cockroach" `
--set Storage="100Gi"
The first node starts up, but the second node crashes with the following:
F191203 17:07:35.042824 214 storage/store_raft.go:322 [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]
Full log:
CockroachDB node starting at 2019-12-03 17:07:34.142182444 +0000 UTC (took 6.1s)
build: CCL v19.2.1 @ 2019/11/18 23:23:55 (go1.12.12)
webui: http://0.0.0.0:8080
sql: postgresql://root@cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257?sslmode=disable
RPC client flags: /cockroach/cockroach <client cmd> --host=cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 --insecure
logs: /cockroach/cockroach-data/logs
temp dir: /cockroach/cockroach-data/cockroach-temp261131495
external I/O path: /cockroach/cockroach-data/extern
store[0]: path=/cockroach/cockroach-data
status: restarted pre-existing node
clusterID: dd364f3f-7900-45f2-9a65-7a105cd7a147
nodeID: 3
I191203 17:07:34.153647 232 server/server_update.go:53 [n3] no need to upgrade, cluster already at the newest version
I191203 17:07:34.191164 234 sql/event_log.go:130 [n3] Event: "node_restart", target: 3, info: {Descriptor:{NodeID:3 Address:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 Attrs: Locality: ServerVersion:19.2 BuildTag:v19.2.1 StartedAt:1575392852575521409 LocalityAddress:[] ClusterName: SQLAddress:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257} ClusterID:dd364f3f-7900-45f2-9a65-7a105cd7a147 StartedAt:1575392852575521409 LastUp:1575392800598285198}
I191203 17:07:34.360523 214 storage/replica_raftstorage.go:794 [n3,s3,r10/2:{-}] applying LEARNER snapshot [id=ea03bb31 index=18]
I191203 17:07:34.487170 214 storage/engine/rocksdb.go:119 [db/version_set.cc:3086] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
I191203 17:07:34.508684 84 gossip/gossip.go:1531 [n3] node has connected to cluster via gossip
I191203 17:07:34.508951 84 storage/stores.go:259 [n3] wrote 2 node addresses to persistent storage
I191203 17:07:34.711691 214 storage/engine/rocksdb.go:119 [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711725 214 storage/engine/rocksdb.go:119 [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711873 214 storage/engine/rocksdb.go:119 [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711893 214 storage/engine/rocksdb.go:119 [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711990 214 storage/replica_raftstorage.go:815 [n3,s3,r10/2:{-}] applied LEARNER snapshot [total=351ms ingestion=4@-9223372036855ms id=ea03bb31 index=18]
E191203 17:07:35.042651 214 util/log/crash_reporting.go:537 [n3,s3,r10/2:{-}] Reported as error 2cfee4bb679a4beb8a489e30d87ab8ea
F191203 17:07:35.042824 214 storage/store_raft.go:322 [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:908
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:526
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1337
while applying snapshot
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:528
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1337
goroutine 214 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000438300, 0xc000438360, 0x0, 0xc0031ba280)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1017 +0xb1
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x7508900, 0xc000000004, 0x6cd15ce, 0x15, 0x142, 0xc00051ac00, 0x1087)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0x95b
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x4a54f40, 0xc003763560, 0xc000000004, 0x3, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x4a54f40, 0xc003763560, 0x2, 0xc000000004, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.FatalfDepth(...)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:199
github.com/cockroachdb/cockroach/pkg/storage.maybeFatalOnRaftReadyErr(0x4a54f40, 0xc003763560, 0x4104925, 0x17, 0x49e0600, 0xc003a98600, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:859 +0x1de
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x4a54f40, 0xc003763560, 0xc003694000, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:322 +0x25f
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000c48300, 0x4a54f40, 0xc003763560, 0xc00077dbc8, 0xc0035b5418, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200 +0x16b
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0xe5401ee131bb03ea, 0x3f9d1493005c399, 0xc000545340, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262 +0x1a4
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0x7f30769bb2c8, 0xc000369540, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854 +0x6d3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1(0x4a54f40, 0xc0035d5bf0, 0xc00054d320, 0x411c3aa)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59 +0x1d9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00054d320, 0x4a54f40, 0xc0035d5bf0, 0x411c3aa, 0x1e, 0xc00331bc30, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc000c48300, 0xc00077db80, 0x7f30769bb298, 0xc000369540, 0xc000369540, 0x30)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49 +0xe6
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x4aa3720, 0xc000369540, 0xc000ca40f0, 0x4a54f40, 0xc0035d5b30, 0x78c3df, 0xc000544ff0)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x4a54f40, 0xc0035d5b30)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00054d320, 0x4a54f40, 0xc0035d5b30, 0xc000075780, 0x32, 0x0, 0x0, 0xc0035d5b60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131
Do I need to specify different options in my storage class?
