8-Bit Floating Point for AI/ML? | Amazon and Microsoft Tech Job Sheds

welcome to long field of view— where I perused the news of the week and stripped it down to the nitty-gritty.let’s work out what is really important.

This week: New ideas bring low-power ML inference to more big-tech jobs.

1. Fast AI with FP8

First up this week is a new proposal for standardizing 8-bit floating point storage. The idea is to speed up real-time inference where speed and power efficiency matter. More important than accuracy..

Analysis: FP8 > INT8

Limiting data to byte width has obvious speed and power advantages. 8 bit integer It has been used for some time in deep learning, but it suffers from overflow problems. Calculating the exponent gives much wider dynamic range.

Karen Heyman: Does floating point 8 solve AI/ML overhead?

“FP32 or FP16 are just unnecessary overhead”
While the media talks about ChatGPT, engineers focus on the hardware challenges of running large-scale language models and other deep learning networks. High on the list of ML essentials is how to run models more efficiently with less power, especially in critical applications such as self-driving cars where latency is critical.
…
IEEE 754, which defines FP32-bit and FP64, was designed for scientific computing where precision is the ultimate consideration. … machine learning often doesn’t need that much accuracy. … Arm, Intel, and Nvidia have … two different flavors of FP8, E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa), to save energy and performance. ) published a white paper proposing overhead.
…
Training requires a wide dynamic range to continuously adjust the coefficients, which is characteristic of backpropagation. …and my guess is that there is a quantization step that shifts the network to FP8 or integer 8 (Int8), since continuing operations in FP32 or FP16 is simply unnecessary overhead. … There is debate as to whether an FP4 standard exists as the quest for lower power continues.

Interesting discussion. As ardit33 understands it:

This is a great article. I recently had to deal with this and FP8/INT8 are very important especially for mobile/on-device inference.

AI will begin to be built into devices of all kinds…and in many cases backend reasoning is undesirable or unfeasible. If the industry moves to his INT8/FP8 inference as the standard, that would be very nice.

Is there another way? Wolfsta:

Personally, I would expect Posit to be more recruited. The main reason is that their behavior is a bit smarter than floats, but there are also advantages associated with using them for AI.

2. Amazon and Microsoft cut 28,000 jobs

Two Washington state giants announced massive job cuts this week.just like them adjusted their news.

Analysis: Welcome to a recession that’s not a recession

We are not in recession by the official definition. But given the behavior of tech companies, it certainly seems so.

Matt Day and Spencer Soper: Amazon preparing for new job cuts

“among some of the big technology companies”
The company announced earlier this month that it would lay off more than 18,000 employees. …they represent about 6% of Amazon’s 350,000 employees.
…
Amazon is one of the big tech companies to cut its ranks, including Cisco Systems, Intel, Metaplatforms, Qualcomm and Salesforce.

However, it doesn’t necessarily affect DevOps jobs. However, theodp doesn’t like optics.

The hiring party at Amazon is over. The number of vacancies in the software development category fell from 32,692 in May 2022 to 299 in January 2023.
…
[Last month] Amazon has delayed the start dates of some college graduates who were supposed to join the company in May 2023, telling students they won’t be able to join until the end of 2023, citing the “macroeconomic environment.”

And the other shoe fell off. Dina Bass joined Matt and Spencer in Round II. Microsoft plans to cut 10,000 jobs

The software giant has started notifying some of the 10,000 employees who will lose their jobs this quarter. …the companies said painful steps were needed to compensate for weak sales and the possibility of a recession that was making customers more wary.

Before the job cuts were announced, Chief Executive Satya Nadella said the tech industry was going through a period of slowing growth and adjustments were needed. with our own technology. ”

Unusually, this time I learned about Microsoft’s job cuts from the press, not from the CEO. Vaast feels weird about it:

We learned about Microsoft’s job cuts in the news yesterday, but at this point no one knows who will be affected. I’m sure they’ll handle it as best they can, but it’s strange to first learn about it from the news instead of the CEO. I received an email from Satya this morning…Most of the companies are currently not hiring, so there are not many internal job openings available for those who have been laid off.
…
A leak is [not] Unusually… Learn This Way feel strange. …At Microsoft, the CEO usually does a great job of announcing important news to employees via email and making it public for investors at the same time. Not this time.

hang on. pause. Are we in recession? nightflameauto mourns the dignity of labor.

i believe the current definition recession, as recently utilized by businesses and governments alike, is “a nebulous conception of a future in which profits are diminishing, lowering the bargaining power of employees, their career interests, and their morale.” We have to fight until they recover.” And stop thinking that your work is of any value to the company you work for. Big companies are basically fed up with employees who brazenly believe they should be paid fairly for their work. The Solution: Constant chatter about the recession while continuing to implement historically horrific policies that destroy employee morale.
…
No, capitalism is great. It smells like death and decay, right?

what’s that? Want to form a trade union? Mabo is not a socialist.

In my opinion Union Not the right word for what I want. I want a co-op where the employee organization owns the company. Profits are a bonus for everyone.
…
I don’t want to be shielded from the company’s goal of making as much money as possible. I want my company’s goal to be employee success.