April 2022 - WLUG - wlug.mailman3.com

Re: Atlassian outage

by John Stoffel

>>>>> "Jared" == Jared Greenwald via WLUG <wlug(a)lists.wlug.org> writes: Jared> At least they're owning the mistake, which is something I guess... Jared> https://www.atlassian.com/engineering/april-2022-outage-update I want people to carefully reade these four paragraphs and ask themselves, what is the difference between restoring one customer quickly using a well tested process, and restoring 400 customers? We also maintain immutable backups that are designed to be resilient against data corruption events, which enable recovery to a previous point in time. Backups are retained for 30 days, and Atlassian continuously tests and audits storage backups for restoration. Using these backups, we regularly roll-back individual customers or a small set of customers who accidentally delete their own data. And, if required, we can restore all customers at once into a new environment. What we have not (yet) automated is restoring a large subset of customers into our existing (and currently in use) environment without affecting any of our other customers. Within our cloud environment, each data store contains data from multiple customers. Because the data deleted in this incident was only a portion of data stores that are continuing to be used by other customers, we have to manually extract and restore individual pieces from our backups. Each customer site recovery is a lengthy and complex process, requiring internal validation and final customer verification when the site is restored. These paragraphs are carefully written to imply one things, but show that they have no system in place for proper rollbacks when an entire customer gets purged. I suspect that their "nuke it all from orbit" script *also* nuked the immutable backups as well, which is why this is all taking so long to restore. I do hope the people who messed up are fired, and if anyone is fired it's the managers who set up this debacle. John

April 13, 2022

Re: Atlassian outage

by John Stoffel

>>>>> "Tim" == Tim Keller via WLUG <wlug(a)lists.wlug.org> writes: Tim> What do people of the crazy atlassian outage? They're talking Tim> about some customers still being offline for two weeks! Live by the cloud, die by the cloud... I think they seriously screwed up, which goes to show you the programming is 10% the work to be done, and 900% error, bounds and exception handling. All of which computers do poorly, but which we humans do reasonably well. But we're *slow* and the cloud is all about speed. In this case, it also shows that Atlassian's DR practices were crap in a major way. And if I was self hosting it internally, and knew that I was being forced to the cloud... I'd be yelling like hell at my sales rep and working to get off Atlassian products. The cloud has some advantages, and is lovely if you have a work load that scales horizontally really well, and which also bursts up in terms of performance need. But when you have a fairly static demand, it's maybe not the best thing to do. But ... the cloud sells itself on doing all those hard things well, like redundancy, resiliency, durability, etc. dual feed UPS, tested generators, reliable AC systems, etc. All the things a good Co-Lo or self hosted people handle to some level of execution. The cloud should (ideally) have all this down to a T. But ... when it doesn't, all hell breaks loose. John

April 13, 2022

Re: Atlassian outage

by Jared Greenwald

At least they're owning the mistake, which is something I guess... https://www.atlassian.com/engineering/april-2022-outage-update On Wed, Apr 13, 2022 at 11:54 AM Jon "maddog" Hall via WLUG < wlug(a)lists.wlug.org> wrote: > I consider a "faulty script" as the worst type of cyber attack. > > As a friend of one once famously said "I will replace you with a very > small shell script" > > They even made a T-shirt out of it. > > > https://www.teepublic.com/t-shirt/3810999-go-away-or-i-will-replace-you-wit… > > md > > On Wed, Apr 13, 2022 at 11:25 AM Tim Keller <turbofx(a)gmail.com> wrote: > >> While they're claiming it's not a cyber crime thing, it looks like a >> single person was able to do this with a faulty script.. that in itself is >> a bit distressing. >> >> It sounds like they've got a serious process problem. >> Tim. >> >> On Wed, Apr 13, 2022 at 10:54 AM Jon "maddog" Hall < >> jon.maddog.hall(a)gmail.com> wrote: >> >>> >What do people of the crazy atlassian outage? >>> >They're talking about some customers still being offline for two weeks! >>> >>> With Easter approaching I might call it "All your eggs in one basket", >>> and no matter how strong, big and well built that basket is we never can be >>> sure there is not an Easter Egg hiding in there, waiting for us. Although >>> Atlassian states it was not a Cyber Security issue, rom a Cyber Security >>> standpoint I will also point out that there are more and more foxes trying >>> to get at the henhouse. >>> >>> md >>> >>> On Wed, Apr 13, 2022 at 10:43 AM Tim Keller via WLUG < >>> wlug(a)lists.wlug.org> wrote: >>> >>>> What do people of the crazy atlassian outage? >>>> They're talking about some customers still being offline for two weeks! >>>> >>>> Later, >>>> Tim. >>>> >>>> -- >>>> I am leery of the allegiances of any politician who refers to their >>>> constituents as "consumers". >>>> _______________________________________________ >>>> WLUG mailing list -- wlug(a)lists.wlug.org >>>> To unsubscribe send an email to wlug-leave(a)lists.wlug.org >>>> Create Account: https://wlug.mailman3.com/accounts/signup/ >>>> Change Settings: >>>> https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ >>>> Web Forum/Archive: >>>> https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/AWD5H… >>>> >>> >> >> -- >> I am leery of the allegiances of any politician who refers to their >> constituents as "consumers". >> > _______________________________________________ > WLUG mailing list -- wlug(a)lists.wlug.org > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > Create Account: https://wlug.mailman3.com/accounts/signup/ > Change Settings: > https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > Web Forum/Archive: > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/26PO4… >

April 13, 2022

Re: Atlassian outage

by Jon "maddog" Hall

I consider a "faulty script" as the worst type of cyber attack. As a friend of one once famously said "I will replace you with a very small shell script" They even made a T-shirt out of it. https://www.teepublic.com/t-shirt/3810999-go-away-or-i-will-replace-you-wit… md On Wed, Apr 13, 2022 at 11:25 AM Tim Keller <turbofx(a)gmail.com> wrote: > While they're claiming it's not a cyber crime thing, it looks like a > single person was able to do this with a faulty script.. that in itself is > a bit distressing. > > It sounds like they've got a serious process problem. > Tim. > > On Wed, Apr 13, 2022 at 10:54 AM Jon "maddog" Hall < > jon.maddog.hall(a)gmail.com> wrote: > >> >What do people of the crazy atlassian outage? >> >They're talking about some customers still being offline for two weeks! >> >> With Easter approaching I might call it "All your eggs in one basket", >> and no matter how strong, big and well built that basket is we never can be >> sure there is not an Easter Egg hiding in there, waiting for us. Although >> Atlassian states it was not a Cyber Security issue, rom a Cyber Security >> standpoint I will also point out that there are more and more foxes trying >> to get at the henhouse. >> >> md >> >> On Wed, Apr 13, 2022 at 10:43 AM Tim Keller via WLUG <wlug(a)lists.wlug.org> >> wrote: >> >>> What do people of the crazy atlassian outage? >>> They're talking about some customers still being offline for two weeks! >>> >>> Later, >>> Tim. >>> >>> -- >>> I am leery of the allegiances of any politician who refers to their >>> constituents as "consumers". >>> _______________________________________________ >>> WLUG mailing list -- wlug(a)lists.wlug.org >>> To unsubscribe send an email to wlug-leave(a)lists.wlug.org >>> Create Account: https://wlug.mailman3.com/accounts/signup/ >>> Change Settings: >>> https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ >>> Web Forum/Archive: >>> https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/AWD5H… >>> >> > > -- > I am leery of the allegiances of any politician who refers to their > constituents as "consumers". >

April 13, 2022

Re: Atlassian outage

by Tim Keller

While they're claiming it's not a cyber crime thing, it looks like a single person was able to do this with a faulty script.. that in itself is a bit distressing. It sounds like they've got a serious process problem. Tim. On Wed, Apr 13, 2022 at 10:54 AM Jon "maddog" Hall < jon.maddog.hall(a)gmail.com> wrote: > >What do people of the crazy atlassian outage? > >They're talking about some customers still being offline for two weeks! > > With Easter approaching I might call it "All your eggs in one basket", and > no matter how strong, big and well built that basket is we never can be > sure there is not an Easter Egg hiding in there, waiting for us. Although > Atlassian states it was not a Cyber Security issue, rom a Cyber Security > standpoint I will also point out that there are more and more foxes trying > to get at the henhouse. > > md > > On Wed, Apr 13, 2022 at 10:43 AM Tim Keller via WLUG <wlug(a)lists.wlug.org> > wrote: > >> What do people of the crazy atlassian outage? >> They're talking about some customers still being offline for two weeks! >> >> Later, >> Tim. >> >> -- >> I am leery of the allegiances of any politician who refers to their >> constituents as "consumers". >> _______________________________________________ >> WLUG mailing list -- wlug(a)lists.wlug.org >> To unsubscribe send an email to wlug-leave(a)lists.wlug.org >> Create Account: https://wlug.mailman3.com/accounts/signup/ >> Change Settings: >> https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ >> Web Forum/Archive: >> https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/AWD5H… >> > -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

April 13, 2022

Re: Atlassian outage

by Jon "maddog" Hall

>What do people of the crazy atlassian outage? >They're talking about some customers still being offline for two weeks! With Easter approaching I might call it "All your eggs in one basket", and no matter how strong, big and well built that basket is we never can be sure there is not an Easter Egg hiding in there, waiting for us. Although Atlassian states it was not a Cyber Security issue, rom a Cyber Security standpoint I will also point out that there are more and more foxes trying to get at the henhouse. md On Wed, Apr 13, 2022 at 10:43 AM Tim Keller via WLUG <wlug(a)lists.wlug.org> wrote: > What do people of the crazy atlassian outage? > They're talking about some customers still being offline for two weeks! > > Later, > Tim. > > -- > I am leery of the allegiances of any politician who refers to their > constituents as "consumers". > _______________________________________________ > WLUG mailing list -- wlug(a)lists.wlug.org > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > Create Account: https://wlug.mailman3.com/accounts/signup/ > Change Settings: > https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > Web Forum/Archive: > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/AWD5H… >

April 13, 2022

Atlassian outage

by Tim Keller

What do people of the crazy atlassian outage? They're talking about some customers still being offline for two weeks! Later, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

April 13, 2022

Reminder! WLUG Meeting April 14th, 2022! Topic: Fotoxx! Presenter: Dick Miller

by Tim Keller

Hey Everybody, We've got a meeting on April 14th at 7pm! The topic is going to be "Fotoxx." This is a professional grade photo management/editing suite developed by Dick Miller. He's going to be showing off it's many amazing features! We'll be meeting at the Hudson Public Library in our usual space. We'll also be meeting online using Jitsi at our usual haunt as well ( https://meet.jit.si/WlugMA) Snacks and Refreshments will be provided and the whole presentation is going to be recorded and uploaded to our youtube channel! Afterwards we'll head off for dinner! Thanks, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

April 12, 2022

Re: April 14th Meeting Announcement! Topic: Fotoxx Presenter: Dick Miller

by John Stoffel

>>>>> "Tim" == Tim Keller via WLUG <wlug(a)lists.wlug.org> writes: Tim> This month Dick Miller will be speaking about the amazing photo Tim> editing software called "Fotoxx". Tim> https://kornelix.net/fotoxx/fotoxx.html Tim> Time: 7pm Tim> Date: April 14th, 2022 Tim> Location: Physical at the Hudson Public Library and virtual in our usual Jitsi haunt. Tim> https://meet.jit.si/WlugMA Tim> We'll be recording the session and posting it to the WLUG youtube channel: Tim> https://www.youtube.com/channel/UCj-5C7Lr4B_2iFr9_vAnmaA Looking forward to being there and seeing this presentation. Would it help if I brought a tablet and pen to be used/tested with the software? As a lefty who can only mouse with the right hand, but *sucks* drawing with the mouse, I'd be interested to see how that works. John

April 7, 2022

April 14th Meeting Announcement! Topic: Fotoxx Presenter: Dick Miller

by Tim Keller

Hey Everybody, This month Dick Miller will be speaking about the amazing photo editing software called "Fotoxx". https://kornelix.net/fotoxx/fotoxx.html From their website: Fotoxx is a free open source *Linux* program for image editing and collection management. The goal is to meet the needs of serious photographers while remaining fast and easy to use. This presentation has been long in the making. Dick reached out to Chuck that due to scheduling conflicts more than a decade ago he was unable to schedule the presentation! Time: 7pm Date: April 14th, 2022 Location: Physical at the Hudson Public Library and virtual in our usual Jitsi haunt. https://meet.jit.si/WlugMA We'll be recording the session and posting it to the WLUG youtube channel: https://www.youtube.com/channel/UCj-5C7Lr4B_2iFr9_vAnmaA As usual, we'll head off after the meeting for dinner and keep the conversation going. I really hope to see you there!! Later, Tim.

April 6, 2022