Oct 5 20:38:08 server-2s07 systemd[1]: ceph-osd@48.service: Service RestartSec=10s expired, scheduling restart.
Oct 5 20:38:08 server-2s07 systemd[1]: ceph-osd@48.service: Scheduled restart job, restart counter is at 2.
Oct 5 20:38:08 server-2s07 systemd[1]: Stopped Ceph object storage daemon osd.48.
Oct 5 20:38:08 server-2s07 systemd[1]: Starting Ceph object storage daemon osd.48...
Oct 5 20:38:08 server-2s07 systemd[1]: Started Ceph object storage daemon osd.48.
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: tcmalloc: large alloc 3342344320348028928 bytes == (nil) @ 0x7ffbdacad760 0x7ffbdaccec64 0x56079e44df8e 0x56079e45d174 0x56079e45d221 0x56079e52fcca 0x56079e530568 0x56079e44e01e 0x56079e44e261 0x56079e395ed6 0x56079e39c35b 0x56079e3e3bc1 0x56079e3e57e9 0x56079e3685ca 0x56079de027e4 0x56079de14c5c 0x56079de6e047 0x56079d993a70 0x56079d8e6901 0x7ffbd8b8c493 0x56079d914e3e
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: terminate called after throwing an instance of 'std::bad_alloc'
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: what(): std::bad_alloc
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: *** Caught signal (Aborted) **
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: in thread 7ffbdc1dbf00 thread_name:ceph-osd
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 1: (()+0x12b20) [0x7ffbd9f3fb20]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 2: (gsignal()+0x10f) [0x7ffbd8ba037f]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 3: (abort()+0x127) [0x7ffbd8b8adb5]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 4: (()+0x9009b) [0x7ffbd955809b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 5: (()+0x9653c) [0x7ffbd955e53c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 6: (()+0x95559) [0x7ffbd955d559]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 7: (__gxx_personality_v0()+0x2a8) [0x7ffbd955ded8]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 8: (()+0x10b03) [0x7ffbd8f3eb03]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 9: (_Unwind_RaiseException()+0x2b1) [0x7ffbd8f3f071]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 10: (__cxa_throw()+0x3b) [0x7ffbd955e7eb]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 11: (()+0x19fa4) [0x7ffbdacacfa4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 12: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7ffbdaccec96]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 13: (()+0x10d0f8e) [0x56079e44df8e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 14: (rocksdb::Version::~Version()+0x104) [0x56079e45d174]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 15: (rocksdb::Version::Unref()+0x21) [0x56079e45d221]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 16: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0x5a) [0x56079e52fcca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 17: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x88) [0x56079e530568]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 18: (rocksdb::VersionSet::~VersionSet()+0x5e) [0x56079e44e01e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 19: (rocksdb::VersionSet::~VersionSet()+0x11) [0x56079e44e261]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 20: (rocksdb::DBImpl::CloseHelper()+0x616) [0x56079e395ed6]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 21: (rocksdb::DBImpl::~DBImpl()+0x83b) [0x56079e39c35b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 22: (rocksdb::DBImplReadOnly::~DBImplReadOnly()+0x11) [0x56079e3e3bc1]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 23: (rocksdb::DB::OpenForReadOnly(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x1089) [0x56079e3e57e9]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x14ca) [0x56079e3685ca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 25: (BlueStore::_open_db(bool, bool, bool)+0x1314) [0x56079de027e4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 26: (BlueStore::_open_db_and_around(bool)+0x4c) [0x56079de14c5c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 27: (BlueStore::_mount(bool, bool)+0x847) [0x56079de6e047]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 28: (OSD::init()+0x380) [0x56079d993a70]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 29: (main()+0x47f1) [0x56079d8e6901]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 30: (__libc_start_main()+0xf3) [0x7ffbd8b8c493]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 31: (_start()+0x2e) [0x56079d914e3e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 2021-10-05T20:38:10.016+0700 7ffbdc1dbf00 -1 *** Caught signal (Aborted) **
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: in thread 7ffbdc1dbf00 thread_name:ceph-osd
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 1: (()+0x12b20) [0x7ffbd9f3fb20]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 2: (gsignal()+0x10f) [0x7ffbd8ba037f]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 3: (abort()+0x127) [0x7ffbd8b8adb5]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 4: (()+0x9009b) [0x7ffbd955809b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 5: (()+0x9653c) [0x7ffbd955e53c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 6: (()+0x95559) [0x7ffbd955d559]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 7: (__gxx_personality_v0()+0x2a8) [0x7ffbd955ded8]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 8: (()+0x10b03) [0x7ffbd8f3eb03]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 9: (_Unwind_RaiseException()+0x2b1) [0x7ffbd8f3f071]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 10: (__cxa_throw()+0x3b) [0x7ffbd955e7eb]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 11: (()+0x19fa4) [0x7ffbdacacfa4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 12: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7ffbdaccec96]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 13: (()+0x10d0f8e) [0x56079e44df8e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 14: (rocksdb::Version::~Version()+0x104) [0x56079e45d174]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 15: (rocksdb::Version::Unref()+0x21) [0x56079e45d221]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 16: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0x5a) [0x56079e52fcca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 17: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x88) [0x56079e530568]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 18: (rocksdb::VersionSet::~VersionSet()+0x5e) [0x56079e44e01e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 19: (rocksdb::VersionSet::~VersionSet()+0x11) [0x56079e44e261]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 20: (rocksdb::DBImpl::CloseHelper()+0x616) [0x56079e395ed6]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 21: (rocksdb::DBImpl::~DBImpl()+0x83b) [0x56079e39c35b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 22: (rocksdb::DBImplReadOnly::~DBImplReadOnly()+0x11) [0x56079e3e3bc1]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 23: (rocksdb::DB::OpenForReadOnly(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x1089) [0x56079e3e57e9]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x14ca) [0x56079e3685ca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 25: (BlueStore::_open_db(bool, bool, bool)+0x1314) [0x56079de027e4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 26: (BlueStore::_open_db_and_around(bool)+0x4c) [0x56079de14c5c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 27: (BlueStore::_mount(bool, bool)+0x847) [0x56079de6e047]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 28: (OSD::init()+0x380) [0x56079d993a70]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 29: (main()+0x47f1) [0x56079d8e6901]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 30: (__libc_start_main()+0xf3) [0x7ffbd8b8c493]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 31: (_start()+0x2e) [0x56079d914e3e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 0> 2021-10-05T20:38:10.016+0700 7ffbdc1dbf00 -1 *** Caught signal (Aborted) **
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: in thread 7ffbdc1dbf00 thread_name:ceph-osd
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 1: (()+0x12b20) [0x7ffbd9f3fb20]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 2: (gsignal()+0x10f) [0x7ffbd8ba037f]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 3: (abort()+0x127) [0x7ffbd8b8adb5]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 4: (()+0x9009b) [0x7ffbd955809b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 5: (()+0x9653c) [0x7ffbd955e53c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 6: (()+0x95559) [0x7ffbd955d559]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 7: (__gxx_personality_v0()+0x2a8) [0x7ffbd955ded8]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 8: (()+0x10b03) [0x7ffbd8f3eb03]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 9: (_Unwind_RaiseException()+0x2b1) [0x7ffbd8f3f071]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 10: (__cxa_throw()+0x3b) [0x7ffbd955e7eb]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 11: (()+0x19fa4) [0x7ffbdacacfa4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 12: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7ffbdaccec96]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 13: (()+0x10d0f8e) [0x56079e44df8e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 14: (rocksdb::Version::~Version()+0x104) [0x56079e45d174]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 15: (rocksdb::Version::Unref()+0x21) [0x56079e45d221]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 16: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0x5a) [0x56079e52fcca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 17: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x88) [0x56079e530568]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 18: (rocksdb::VersionSet::~VersionSet()+0x5e) [0x56079e44e01e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 19: (rocksdb::VersionSet::~VersionSet()+0x11) [0x56079e44e261]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 20: (rocksdb::DBImpl::CloseHelper()+0x616) [0x56079e395ed6]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 21: (rocksdb::DBImpl::~DBImpl()+0x83b) [0x56079e39c35b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 22: (rocksdb::DBImplReadOnly::~DBImplReadOnly()+0x11) [0x56079e3e3bc1]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 23: (rocksdb::DB::OpenForReadOnly(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x1089) [0x56079e3e57e9]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x14ca) [0x56079e3685ca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 25: (BlueStore::_open_db(bool, bool, bool)+0x1314) [0x56079de027e4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 26: (BlueStore::_open_db_and_around(bool)+0x4c) [0x56079de14c5c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 27: (BlueStore::_mount(bool, bool)+0x847) [0x56079de6e047]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 28: (OSD::init()+0x380) [0x56079d993a70]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 29: (main()+0x47f1) [0x56079d8e6901]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 30: (__libc_start_main()+0xf3) [0x7ffbd8b8c493]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 31: (_start()+0x2e) [0x56079d914e3e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 0> 2021-10-05T20:38:10.016+0700 7ffbdc1dbf00 -1 *** Caught signal (Aborted) **
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: in thread 7ffbdc1dbf00 thread_name:ceph-osd
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: ceph version 15.2.14 (cd3bb7e87a2f62c1b862ff3fd8b1eec13391a5be) octopus (stable)
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 1: (()+0x12b20) [0x7ffbd9f3fb20]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 2: (gsignal()+0x10f) [0x7ffbd8ba037f]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 3: (abort()+0x127) [0x7ffbd8b8adb5]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 4: (()+0x9009b) [0x7ffbd955809b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 5: (()+0x9653c) [0x7ffbd955e53c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 6: (()+0x95559) [0x7ffbd955d559]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 7: (__gxx_personality_v0()+0x2a8) [0x7ffbd955ded8]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 8: (()+0x10b03) [0x7ffbd8f3eb03]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 9: (_Unwind_RaiseException()+0x2b1) [0x7ffbd8f3f071]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 10: (__cxa_throw()+0x3b) [0x7ffbd955e7eb]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 11: (()+0x19fa4) [0x7ffbdacacfa4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 12: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x146) [0x7ffbdaccec96]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 13: (()+0x10d0f8e) [0x56079e44df8e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 14: (rocksdb::Version::~Version()+0x104) [0x56079e45d174]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 15: (rocksdb::Version::Unref()+0x21) [0x56079e45d221]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 16: (rocksdb::ColumnFamilyData::~ColumnFamilyData()+0x5a) [0x56079e52fcca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 17: (rocksdb::ColumnFamilySet::~ColumnFamilySet()+0x88) [0x56079e530568]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 18: (rocksdb::VersionSet::~VersionSet()+0x5e) [0x56079e44e01e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 19: (rocksdb::VersionSet::~VersionSet()+0x11) [0x56079e44e261]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 20: (rocksdb::DBImpl::CloseHelper()+0x616) [0x56079e395ed6]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 21: (rocksdb::DBImpl::~DBImpl()+0x83b) [0x56079e39c35b]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 22: (rocksdb::DBImplReadOnly::~DBImplReadOnly()+0x11) [0x56079e3e3bc1]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 23: (rocksdb::DB::OpenForReadOnly(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x1089) [0x56079e3e57e9]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x14ca) [0x56079e3685ca]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 25: (BlueStore::_open_db(bool, bool, bool)+0x1314) [0x56079de027e4]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 26: (BlueStore::_open_db_and_around(bool)+0x4c) [0x56079de14c5c]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 27: (BlueStore::_mount(bool, bool)+0x847) [0x56079de6e047]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 28: (OSD::init()+0x380) [0x56079d993a70]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 29: (main()+0x47f1) [0x56079d8e6901]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 30: (__libc_start_main()+0xf3) [0x7ffbd8b8c493]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: 31: (_start()+0x2e) [0x56079d914e3e]
Oct 5 20:38:10 server-2s07 ceph-osd[255813]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 5 20:38:10 server-2s07 systemd[1]: Started Process Core Dump (PID 255865/UID 0).
Oct 5 20:38:10 server-2s07 systemd[1]: ceph-osd@48.service: Main process exited, code=dumped, status=6/ABRT
Oct 5 20:38:10 server-2s07 systemd[1]: ceph-osd@48.service: Failed with result 'core-dump'.
Oct 5 20:38:11 server-2s07 systemd-coredump[255866]: Process 255813 (ceph-osd) of user 167 dumped core.#012#012Stack trace of thread 255813:#012#0 0x00007ffbd9f3f9bf raise (libpthread.so.0)#012#1 0x000056079df71da3 _ZL19handle_fatal_signali (ceph-osd)#012#2 0x00007ffbd9f3fb20 __restore_rt (libpthread.so.0)#012#3 0x00007ffbd8ba037f raise (libc.so.6)#012#4 0x00007ffbd8b8adb5 abort (libc.so.6)#012#5 0x00007ffbd955809b _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold.1 (libstdc++.so.6)#012#6 0x00007ffbd955e53c _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6)#012#7 0x00007ffbd955d559 __cxa_call_terminate (libstdc++.so.6)#012#8 0x00007ffbd955ded8 __gxx_personality_v0 (libstdc++.so.6)#012#9 0x00007ffbd8f3eb03 _Unwind_RaiseException_Phase2 (libgcc_s.so.1)#012#10 0x00007ffbd8f3f071 _Unwind_RaiseException (libgcc_s.so.1)#012#11 0x00007ffbd955e7eb __cxa_throw (libstdc++.so.6)#012#12 0x00007ffbdacacfa4 _ZN12_GLOBAL__N_110handle_oomEPFPvS0_ES0_bb (libtcmalloc.so.4)#012#13 0x00007ffbdaccec96 _ZN8tcmalloc27allocate_full_cpp_throw_oomEm (libtcmalloc.so.4)#012#14 0x000056079e44df8e _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.constprop.1516 (ceph-osd)#012#15 0x000056079e45d174 _ZN7rocksdb7VersionD1Ev (ceph-osd)#012#16 0x000056079e45d221 _ZN7rocksdb7Version5UnrefEv (ceph-osd)#012#17 0x000056079e52fcca _ZN7rocksdb16ColumnFamilyDataD1Ev (ceph-osd)#012#18 0x000056079e530568 _ZN7rocksdb15ColumnFamilySetD1Ev (ceph-osd)#012#19 0x000056079e44e01e _ZN7rocksdb10VersionSetD1Ev (ceph-osd)#012#20 0x000056079e44e261 _ZN7rocksdb10VersionSetD0Ev (ceph-osd)#012#21 0x000056079e395ed6 _ZN7rocksdb6DBImpl11CloseHelperEv (ceph-osd)#012#22 0x000056079e39c35b _ZN7rocksdb6DBImplD1Ev (ceph-osd)#012#23 0x000056079e3e3bc1 _ZN7rocksdb14DBImplReadOnlyD0Ev (ceph-osd)#012#24 0x000056079e3e57e9 _ZN7rocksdb2DB15OpenForReadOnlyERKNS_9DBOptionsERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS_22ColumnFamilyDescriptorESaISD_EEPSC_IPNS_18ColumnFamilyHandleESaISJ_EEPPS0_b (ceph-osd)#012#25 0x000056079e3685ca _ZN12RocksDBStore7do_openERSobbPKSt6vectorIN10KeyValueDB12ColumnFamilyESaIS3_EE (ceph-osd)#012#26 0x000056079de027e4 _ZN9BlueStore8_open_dbEbbb (ceph-osd)#012#27 0x000056079de14c5c _ZN9BlueStore19_open_db_and_aroundEb (ceph-osd)#012#28 0x000056079de6e047 _ZN9BlueStore6_mountEbb (ceph-osd)#012#29 0x000056079d993a70 _ZN3OSD4initEv (ceph-osd)#012#30 0x000056079d8e6901 main (ceph-osd)#012#31 0x00007ffbd8b8c493 __libc_start_main (libc.so.6)#012#32 0x000056079d914e3e _start (ceph-osd)#012#012Stack trace of thread 255814:#012#0 0x00007ffbd9f3b2fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007ffbd95848f0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6)#012#2 0x000056079e168d37 _ZN4ceph7logging3Log5entryEv (ceph-osd)#012#3 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#4 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255822:#012#0 0x00007ffbd9f3b64a pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000056079dfeab3d _ZN4ceph6common24CephContextServiceThread5entryEv (ceph-osd)#012#2 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#3 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255823:#012#0 0x00007ffbd8c5aa41 __poll (libc.so.6)#012#1 0x000056079dfcaa32 _ZN11AdminSocket5entryEv (ceph-osd)#012#2 0x00007ffbd958aba3 execute_native_thread_routine (libstdc++.so.6)#012#3 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#4 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255834:#012#0 0x00007ffbd8c31d98 __nanosleep (libc.so.6)#012#1 0x00007ffbd8c5db28 usleep (libc.so.6)#012#2 0x000056079dc45b73 _ZN22OpHistoryServiceThread5entryEv (ceph-osd)#012#3 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#4 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255831:#012#0 0x00007ffbd8c5aa41 __poll (libc.so.6)#012#1 0x000056079df72e4b _ZN13SignalHandler5entryEv (ceph-osd)#012#2 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#3 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255815:#012#0 0x00007ffbd8c660f7 epoll_wait (libc.so.6)#012#1 0x000056079e14e424 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (ceph-osd)#012#2 0x000056079e143d8b _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (ceph-osd)#012#3 0x000056079e149f1c _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEjEUlvE_E9_M_invokeERKSt9_Any_data (ceph-osd)#012#4 0x00007ffbd958aba3 execute_native_thread_routine (libstdc++.so.6)#012#5 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#6 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255836:#012#0 0x00007ffbd9f3e65d __lll_lock_wait (libpthread.so.0)#012#1 0x00007ffbd9f37979 __pthread_mutex_lock (libpthread.so.0)#012#2 0x000056079dfbc75e _ZN9SafeTimer12timer_threadEv (ceph-osd)#012#3 0x000056079dfbdea1 _ZN15SafeTimerThread5entryEv (ceph-osd)#012#4 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#5 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255837:#012#0 0x00007ffbd9f3b2fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007ffbd95848f0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6)#012#2 0x000056079dfbc915 _ZN9SafeTimer12timer_threadEv (ceph-osd)#012#3 0x000056079dfbdea1 _ZN15SafeTimerThread5entryEv (ceph-osd)#012#4 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#5 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255816:#012#0 0x00007ffbd8c660f7 epoll_wait (libc.so.6)#012#1 0x000056079e14e424 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (ceph-osd)#012#2 0x000056079e143d8b _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (ceph-osd)#012#3 0x000056079e149f1c _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEjEUlvE_E9_M_invokeERKSt9_Any_data (ceph-osd)#012#4 0x00007ffbd958aba3 execute_native_thread_routine (libstdc++.so.6)#012#5 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#6 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255817:#012#0 0x00007ffbd8c660f7 epoll_wait (libc.so.6)#012#1 0x000056079e14e424 _ZN11EpollDriver10event_waitERSt6vectorI14FiredFileEventSaIS1_EEP7timeval (ceph-osd)#012#2 0x000056079e143d8b _ZN11EventCenter14process_eventsEjPNSt6chrono8durationImSt5ratioILl1ELl1000000000EEEE (ceph-osd)#012#3 0x000056079e149f1c _ZNSt17_Function_handlerIFvvEZN12NetworkStack10add_threadEjEUlvE_E9_M_invokeERKSt9_Any_data (ceph-osd)#012#4 0x00007ffbd958aba3 execute_native_thread_routine (libstdc++.so.6)#012#5 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#6 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255838:#012#0 0x00007ffbd9f3b2fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007ffbd95848f0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6)#012#2 0x000056079dfbc915 _ZN9SafeTimer12timer_threadEv (ceph-osd)#012#3 0x000056079dfbdea1 _ZN15SafeTimerThread5entryEv (ceph-osd)#012#4 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#5 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255835:#012#0 0x00007ffbd9f3b2fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007ffbd95848f0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6)#012#2 0x000056079dc77857 _ZN4ceph5timerINS_11time_detail17coarse_mono_clockEE12timer_threadEv (ceph-osd)#012#3 0x00007ffbd958aba3 execute_native_thread_routine (libstdc++.so.6)#012#4 0x00007ffbd9f3514a start_thread (libpthread.so.0)#012#5 0x00007ffbd8c65dc3 __clone (libc.so.6)#012#012Stack trace of thread 255839:#012#0 0x00007ffbd9f3b2fc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x00007ffbd95848f0 _ZNSt18condition_variable4waitERSt11unique_lockISt5mutexE (libstdc++.so.6)#012#2 0x000056079dfbc915 _ZN9SafeTimer12timer_threadEv (ceph-osd)#012#3 0x000056079dfbdea1 _ZN15SafeTimerThread5entryEv (ceph-osd)#012#
Oct 5 20:38:11 server-2s07 systemd[1]: systemd-coredump@8-255865-0.service: Succeeded.
Oct 5 20:38:18 server-2s07 winbindd[3512]: [2021/10/05 20:38:18.826337, 0] ../../source3/winbindd/winbindd_util.c:747(rescan_forest_trusts)
Oct 5 20:38:18 server-2s07 winbindd[3512]: rescan_forest_trusts: add_trusted_domain: NT_STATUS_NO_SUCH_DOMAIN
Oct 5 20:38:20 server-2s07 systemd[1]: ceph-osd@48.service: Service RestartSec=10s expired, scheduling restart.
Oct 5 20:38:20 server-2s07 systemd[1]: ceph-osd@48.service: Scheduled restart job, restart counter is at 3.
Oct 5 20:38:20 server-2s07 systemd[1]: Stopped Ceph object storage daemon osd.48.
Oct 5 20:38:20 server-2s07 systemd[1]: ceph-osd@48.service: Start request repeated too quickly.
Oct 5 20:38:20 server-2s07 systemd[1]: ceph-osd@48.service: Failed with result 'core-dump'.
Oct 5 20:38:20 server-2s07 systemd[1]: Failed to start Ceph object storage daemon osd.48.
