Is my Ubuntu/Canonical Openstack Disk Architecture Legitimate?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
I have a couple of the new R740 reference spec machines with 1.6TB NMVe to add to the cluster.
Time to adjust my disks:
Old Machines:
R740, 2X10 core Xeon, 384GB RAM, 4X240GB SSD, 6X4TB SATA, 2X10Gbps Copper bonded
New Machines:
R740, 2X12 core 6130 Xeon Gold, 512GB RAM, 2X240GB SSD, 6X4TB SATA, 4X10Gbps Copper bonded, 1.6TB NVMe card
Proposed Disk Architecture:
1. OS - 240GB Usable, 2XSSD RAID1 Mirror (RAID Controller)
2. Hypervisors:
1. LXD - 8TB Usable (ZFS) RAIDz1 Dedupe (zlib2) & Compression, 3XSATA RAW + 1XSSD/240GBXNMVe ARC2
2. KVM, 3XSATA LVM Stripe+Parity
3. Ceph Storage - 72TB, 36TB Usable (2 replicas), 3xSATA JBOD + 1XSSD OSD/240GB NVMe
4. 1.1TB NVMe remaining space (Docker host shared drives, SSD Cinder Volumes, Special Use Cases)
My previous R740 machines used the nasty little BOSS SATA3 SSD cards with 2X240GB Intel (slow but lots of spare space for wear and tear)
Whenever we go to the data center, I am looking at R740XD, which is not Canonical reference spec. 24X1TB SSD all flash storage would be easier to allocate though.
I am doing it this way because I have not found any data on how to just convert the entire machine’s storage to tiered (SSD caches + SATA) ZFS with Deduplication (zlib2) + Compression) which would essentially make each machine a Netapp and with dedupe running I would probably never run out of space, ever.
Here is why:
Images = VM duplicates = Snapshot duplicates
With Cinder, Ceph, LXD/KVM, and Glance running on every machine, everything is a dupe. I am not doing that b/c each charm says “development only” for setting a place on the local filesystem of a local directory as a volume in the disk settings.
Originally I wanted to just mount a big ZFS array (3X4TBSATA Raidz1 + 3X4TBSATA Raidz1 + NVMe ARC2 SSD Cache) at /var/lib to dedupe everything, with separate OSD (RAID0/NVME partition) and OS (RAID1) SSD arrays on the side.
I ruined my home lab machine by trying to use LVM2 SSD caching (it breaks startup) and still have not figured out the bcache thing.
Is this right? I still haven't gotten approval to purchase support licenses for my lab so I am still having to "wing it."
ssd maas openstack zfs nvme
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
add a comment |
I have a couple of the new R740 reference spec machines with 1.6TB NMVe to add to the cluster.
Time to adjust my disks:
Old Machines:
R740, 2X10 core Xeon, 384GB RAM, 4X240GB SSD, 6X4TB SATA, 2X10Gbps Copper bonded
New Machines:
R740, 2X12 core 6130 Xeon Gold, 512GB RAM, 2X240GB SSD, 6X4TB SATA, 4X10Gbps Copper bonded, 1.6TB NVMe card
Proposed Disk Architecture:
1. OS - 240GB Usable, 2XSSD RAID1 Mirror (RAID Controller)
2. Hypervisors:
1. LXD - 8TB Usable (ZFS) RAIDz1 Dedupe (zlib2) & Compression, 3XSATA RAW + 1XSSD/240GBXNMVe ARC2
2. KVM, 3XSATA LVM Stripe+Parity
3. Ceph Storage - 72TB, 36TB Usable (2 replicas), 3xSATA JBOD + 1XSSD OSD/240GB NVMe
4. 1.1TB NVMe remaining space (Docker host shared drives, SSD Cinder Volumes, Special Use Cases)
My previous R740 machines used the nasty little BOSS SATA3 SSD cards with 2X240GB Intel (slow but lots of spare space for wear and tear)
Whenever we go to the data center, I am looking at R740XD, which is not Canonical reference spec. 24X1TB SSD all flash storage would be easier to allocate though.
I am doing it this way because I have not found any data on how to just convert the entire machine’s storage to tiered (SSD caches + SATA) ZFS with Deduplication (zlib2) + Compression) which would essentially make each machine a Netapp and with dedupe running I would probably never run out of space, ever.
Here is why:
Images = VM duplicates = Snapshot duplicates
With Cinder, Ceph, LXD/KVM, and Glance running on every machine, everything is a dupe. I am not doing that b/c each charm says “development only” for setting a place on the local filesystem of a local directory as a volume in the disk settings.
Originally I wanted to just mount a big ZFS array (3X4TBSATA Raidz1 + 3X4TBSATA Raidz1 + NVMe ARC2 SSD Cache) at /var/lib to dedupe everything, with separate OSD (RAID0/NVME partition) and OS (RAID1) SSD arrays on the side.
I ruined my home lab machine by trying to use LVM2 SSD caching (it breaks startup) and still have not figured out the bcache thing.
Is this right? I still haven't gotten approval to purchase support licenses for my lab so I am still having to "wing it."
ssd maas openstack zfs nvme
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago
add a comment |
I have a couple of the new R740 reference spec machines with 1.6TB NMVe to add to the cluster.
Time to adjust my disks:
Old Machines:
R740, 2X10 core Xeon, 384GB RAM, 4X240GB SSD, 6X4TB SATA, 2X10Gbps Copper bonded
New Machines:
R740, 2X12 core 6130 Xeon Gold, 512GB RAM, 2X240GB SSD, 6X4TB SATA, 4X10Gbps Copper bonded, 1.6TB NVMe card
Proposed Disk Architecture:
1. OS - 240GB Usable, 2XSSD RAID1 Mirror (RAID Controller)
2. Hypervisors:
1. LXD - 8TB Usable (ZFS) RAIDz1 Dedupe (zlib2) & Compression, 3XSATA RAW + 1XSSD/240GBXNMVe ARC2
2. KVM, 3XSATA LVM Stripe+Parity
3. Ceph Storage - 72TB, 36TB Usable (2 replicas), 3xSATA JBOD + 1XSSD OSD/240GB NVMe
4. 1.1TB NVMe remaining space (Docker host shared drives, SSD Cinder Volumes, Special Use Cases)
My previous R740 machines used the nasty little BOSS SATA3 SSD cards with 2X240GB Intel (slow but lots of spare space for wear and tear)
Whenever we go to the data center, I am looking at R740XD, which is not Canonical reference spec. 24X1TB SSD all flash storage would be easier to allocate though.
I am doing it this way because I have not found any data on how to just convert the entire machine’s storage to tiered (SSD caches + SATA) ZFS with Deduplication (zlib2) + Compression) which would essentially make each machine a Netapp and with dedupe running I would probably never run out of space, ever.
Here is why:
Images = VM duplicates = Snapshot duplicates
With Cinder, Ceph, LXD/KVM, and Glance running on every machine, everything is a dupe. I am not doing that b/c each charm says “development only” for setting a place on the local filesystem of a local directory as a volume in the disk settings.
Originally I wanted to just mount a big ZFS array (3X4TBSATA Raidz1 + 3X4TBSATA Raidz1 + NVMe ARC2 SSD Cache) at /var/lib to dedupe everything, with separate OSD (RAID0/NVME partition) and OS (RAID1) SSD arrays on the side.
I ruined my home lab machine by trying to use LVM2 SSD caching (it breaks startup) and still have not figured out the bcache thing.
Is this right? I still haven't gotten approval to purchase support licenses for my lab so I am still having to "wing it."
ssd maas openstack zfs nvme
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
I have a couple of the new R740 reference spec machines with 1.6TB NMVe to add to the cluster.
Time to adjust my disks:
Old Machines:
R740, 2X10 core Xeon, 384GB RAM, 4X240GB SSD, 6X4TB SATA, 2X10Gbps Copper bonded
New Machines:
R740, 2X12 core 6130 Xeon Gold, 512GB RAM, 2X240GB SSD, 6X4TB SATA, 4X10Gbps Copper bonded, 1.6TB NVMe card
Proposed Disk Architecture:
1. OS - 240GB Usable, 2XSSD RAID1 Mirror (RAID Controller)
2. Hypervisors:
1. LXD - 8TB Usable (ZFS) RAIDz1 Dedupe (zlib2) & Compression, 3XSATA RAW + 1XSSD/240GBXNMVe ARC2
2. KVM, 3XSATA LVM Stripe+Parity
3. Ceph Storage - 72TB, 36TB Usable (2 replicas), 3xSATA JBOD + 1XSSD OSD/240GB NVMe
4. 1.1TB NVMe remaining space (Docker host shared drives, SSD Cinder Volumes, Special Use Cases)
My previous R740 machines used the nasty little BOSS SATA3 SSD cards with 2X240GB Intel (slow but lots of spare space for wear and tear)
Whenever we go to the data center, I am looking at R740XD, which is not Canonical reference spec. 24X1TB SSD all flash storage would be easier to allocate though.
I am doing it this way because I have not found any data on how to just convert the entire machine’s storage to tiered (SSD caches + SATA) ZFS with Deduplication (zlib2) + Compression) which would essentially make each machine a Netapp and with dedupe running I would probably never run out of space, ever.
Here is why:
Images = VM duplicates = Snapshot duplicates
With Cinder, Ceph, LXD/KVM, and Glance running on every machine, everything is a dupe. I am not doing that b/c each charm says “development only” for setting a place on the local filesystem of a local directory as a volume in the disk settings.
Originally I wanted to just mount a big ZFS array (3X4TBSATA Raidz1 + 3X4TBSATA Raidz1 + NVMe ARC2 SSD Cache) at /var/lib to dedupe everything, with separate OSD (RAID0/NVME partition) and OS (RAID1) SSD arrays on the side.
I ruined my home lab machine by trying to use LVM2 SSD caching (it breaks startup) and still have not figured out the bcache thing.
Is this right? I still haven't gotten approval to purchase support licenses for my lab so I am still having to "wing it."
ssd maas openstack zfs nvme
ssd maas openstack zfs nvme
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
asked Mar 21 at 15:16
spyderdynespyderdyne
512519
512519
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago
add a comment |
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1127565%2fis-my-ubuntu-canonical-openstack-disk-architecture-legitimate%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1127565%2fis-my-ubuntu-canonical-openstack-disk-architecture-legitimate%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I am attempting to feed the proposed ZFS settings via User Scripts in MaaS, and will then need to attach the cache drives to the ZFS array by a manual cloud-nit step I guess. The issue is with using ZFS to replace the /var filesystem, since that is where the money gets made. WIll update if I ever get past "Commissioning."
– spyderdyne
2 days ago