About Robin Edgar

Organisational Structures | Technology and Science | Military, IT and Lifestyle consultancy | Social, Broadcast & Cross Media | Flying aircraft

Automakers Can’t Give Up The Idea Of Turning Everyday Features Into Subscription Services With Fees

At the same time car companies are fighting the right to repair movement (and the state and federal legislation popping up everywhere), they’re continuing the quest to turn everyday features — like heated seats — into something users have to pay a recurring fee for.

In 2019, BMW had to abandon a plan to charge $80 per year for Apple CarPlay. The company, having learned nothing, began floating the idea of charging a subscription for features back in 2020, when it proposed making heated seats and heated steering wheels something you pay a permanent monthly fee for. Last December, Toyota proposed imposing a monthly fee for customers who wanted to be able to remotely start their vehicles.

Each and every time these proposals come forward the consumer response is swift and overwhelmingly negative. But with $20 billion in annual additional potential revenue on the table between now and 2030, the industry seems poised to ignore consumers:

“Still, automakers see dollar signs. Stellantis (formerly Fiat Chrysler), Ford, and GM each aim to generate at least $20 billion in annual revenue from software services by 2030. Over-the-air capabilities open up huge opportunities for carmakers to introduce new subscription or pay-per use features over time, Wakefield, of AlixPartners, said. Someday, you may be able to fork over extra to make your car more efficient, sportier, or — in an electric vehicle — unlock extra range for road trips.”

Keep in mind these are decisions being made during a pandemic when most households continue to struggle.

This sort of nickel-and-diming works well in the telecom sector where captive subscribers often can’t switch to a different competitor. But in the auto space, companies risk opening the door to competitors gaining inroads by… not being nickel-and-diming assholes. Many companies may also be overestimating their own product quality; one JD Power survey found that 58% of people who use an automaker’s smartphone app wouldn’t be willing to pay for it. At the same time, as with gaming microtransactions, if enough people are willing to pay to make it worth it, it may not matter what the majority of car consumers think.

Source: Automakers Can’t Give Up The Idea Of Turning Everyday Features Into Subscription Services With Fees | Techdirt

Saturn’s high-altitude winds generate an extraordinary aurorae, tells us more about what they are

Leicester space scientists have discovered a never-before-seen mechanism fuelling huge planetary aurorae at Saturn.

Saturn is unique among planets observed to date in that some of its aurorae are generated by swirling winds within its own atmosphere, and not just from the planet’s surrounding magnetosphere.

At all other observed planets, including Earth, aurorae are only formed by powerful currents that flow into the planet’s atmosphere from the surrounding magnetosphere. These are driven by either interaction with charged particles from the Sun (as at the Earth) or volcanic material erupted from a moon orbiting the planet (as at Jupiter and Saturn).

This discovery changes scientists’ understanding of planetary aurorae and answers one of the first mysteries raised by NASA’s Cassini probe, which reached Saturn in 2004: why can’t we easily measure the length of a day on the Ringed Planet?

When it first arrived at Saturn, Cassini tried to measure the bulk rotation rate of the planet, that determines the length of its day, by tracking ‘pulses’ from Saturn’s atmosphere. To the great surprise of those making the measurements, they found that the rate appeared to have changed over the two decades since the last spacecraft to have flown past the planet—Voyager 2, also operated by NASA—in 1981.

Leicester Ph.D. researcher Nahid Chowdhury is a member of the Planetary Science Group within the School of Physics and Astronomy and corresponding author for the study, published in Geophysical Research Letters. He said:

“Saturn’s internal rotation rate has to be constant, but for decades researchers have shown that numerous periodic properties related to the planet—the very measurements we’ve used at other planets to understand the internal rotation rate, such as the radio emission—tend to change with time. What’s more, there are also independent periodic features seen in the northern and southern hemispheres which themselves vary over the course of a season on the planet.

“Our understanding of the physics of planetary interiors tells us the true rotation rate of the planet can’t change this quickly, so something unique and strange must be happening at Saturn. Several theories have been touted since the advent of the NASA Cassini mission trying to explain the mechanism/s behind these observed periodicities. This study represents the first detection of the fundamental driver, situated in the upper atmosphere of the planet, which goes on to generate both the observed planetary periodicities and aurorae.

Simplified figure showing the direction of winds within layers of Saturn’s atmosphere. Credit: Nahid Chowdhury/University of Leicester

“It’s absolutely thrilling to be able to provide an answer to one of the longest standing questions in our field. This is likely to initiate some rethinking about how local atmospheric weather effects on a planet impact the creation of aurorae, not just in our own Solar System but farther afield too.”

[…]

They measured infrared emissions from the gas giant’s upper atmosphere using the Keck Observatory in Hawai’i and mapped the varying flows of Saturn’s ionosphere, far below the magnetosphere, over the course of a month in 2017.

This map, when fixed against the known pulse of Saturn’s radio aurorae, showed that a significant proportion of the planet’s aurorae are generated by the swirling pattern of weather in its atmosphere and are responsible for the planet’s observed variable rate of rotation.

Researchers believe the system is driven by energy from Saturn’s thermosphere, with winds in the ionosphere observed between 0.3 and 3.0 kilometres per second.

[…]

recently, many researchers have focused on the possibility that it is Saturn’s upper atmosphere that causes this variability.

“This search for a new type of aurora harks back to some of the earliest theories about Earth’s aurora. We now know that aurorae on Earth are powered by interactions with the stream of charged particles driven from the Sun. But I love that the name Aurora Borealis originates from the ‘the Dawn of the Northern Wind’. These observations have revealed that Saturn has a true Aurora Borealis—the first ever aurora driven by the winds in the atmosphere of a planet.”

Dr. Kevin Baines, a JPL-Caltech-based co-author of the study and a member of the Cassini Science Team, added:

“Our study, by conclusively determining the origin of the mysterious variability in radio pulses, eliminates much of the confusion into Saturn’s bulk rotation rate and the length of the day on Saturn.”

Because of the variable rotation rates observed at Saturn, scientists have been prevented from using the regular pulse of radio emission to calculate the bulk internal rotation rate. Fortunately, a novel method was developed by Cassini scientists using gravity-induced perturbations in Saturn’s complex ring system, which now seems to be the most accurate means of measuring the planet’s bulk rotational period, which was determined in 2019 to be 10 hours, 33 minutes and 38 seconds.

[…]

 

Source: Saturn’s high-altitude winds generate an extraordinary aurorae, study finds

Bitcoin a lifeline for sex workers, like ex-nurse making $1.3 million – banks and other payment providers refuse to process them

[…]

Knox describes herself as “one of the most outspoken sex workers, particularly for crypto.” Her interest kicked off in 2014, which is when she says several vendors, including PayPal, Square Cash, and Venmo, shut down her accounts because of red flags related to sex work.

So Knox started accepting cryptocurrencies instead. Her first exchange of bitcoin for content was pretty casual.

It started on a Skype call with a client. “I had a Coinbase account at the time, and he said, ‘Hold your QR code right to this camera here,’ and he sent it through the camera. And I got it,” she explained.

It took 15 minutes, and there were no chargebacks, no website commission fees, and no bank intermediaries to turn down the transaction – all major pluses in her industry. But the biggest attraction was having total and irreversible ownership over the money she had earned.

[…]

“The majority of sex work in the U.S. is legal. It’s not dealt with fairly, but it’s still legal,” explained Kristen DiAngelo, an activist and Sacramento-based sex worker who has spent over four decades in the industry. “Stripping is legal…massage is legal…escorting is legal. The only thing that’s really illegal in the U.S. is the honest exchange of sexual activity for remuneration, for money.”

Some escorts – who charge anywhere from $1,700 an hour to $11,000 for a full 24 hours – now explicitly say in their ads that they prefer to be paid in bitcoin or ethereum.

[…]

Allie Rae is a 37-year-old mother of three boys who says she went from making about $84,000 a year as an ICU nurse in Boston to $1.3 million, thanks to her work on OnlyFans, which has more than 130 million users.

[…]

DiAngelo tells CNBC she will never forget the first time her bank account was closed without warning.

It happened when she was on a trip to Washington, D.C. over a decade ago.

“I had just gone into the bank, made a deposit, and I went to buy lunch in Dupont Circle,” said DiAngelo. “I gave him my card, and it was declined. I gave him my card, and it was declined again. And I gave my card again, and it was declined again. And I was like, ‘No, no, no, no, that can’t be right. There’s something wrong.’”

DiAngelo called Citibank and learned that her account had been frozen and she should tear up her credit card. DiAngelo says the customer service rep told her that they weren’t “at liberty” to tell her why it had happened, and she would have to write a formal letter to request additional details.

They did, however, say that she was still responsible for any money owed.

[…]

So DiAngelo did what other sex workers do: She “platform hopped,” meaning that she brought her money to another bank. When they also flagged and closed her account, she moved on to the next. After being shut out of a third bank, DiAngelo says she turned exclusively to bitcoin for her online banking needs.

Nearly every sex worker interviewed for this story mentioned platform hopping. The government has a set of anti-trafficking guidelines drawn up by the Financial Crimes Enforcement Network, or FinCEN, and the banks and big payment apps keep an eye out for activity deemed suspicious by those guidelines. Those red flags include making cash deposits frequently – a hallmark of the sex work profession.

[…]

In 2014, for example, PayPal booted her because of a payment for her used socks that was large enough to get red-flagged. Knox says neither she nor the buyer were refunded. (PayPal tells CNBC that her account was “closed due to policy violations.”)

Later, in 2016, Coinbase closed her account and blocked her from making others. (Coinbase acknowledged to CNBC that its terms of service prohibit the use of its “commerce or retail services connected to adult content.”)

“We’re the ones being punished – not the traffickers, not those that are actually abusing workers,” said Alana Evans, who has been an adult performer since the late 90′s. Evans is currently president of the Adult Performance Artists Guild, or APAG, a federally recognized union within the adult industry that represents all workers from adult film set actors, to content creators.

“They’ve attacked our banking; our ability to operate like the rest of the world,” explained DiAngelo. “You don’t exist if you can’t use the banking system.”

[…]

One hazard of the trade are chargebacks, in which a transaction is reversed when a consumer claims they have been fraudulently charged for a good or service they did not receive. It is a tool designed to protect consumers, but many sex workers say it is a tool that is abused in their industry by clients who dispute a transaction for a product or service they have already received.

Take OnlyFans. There are some customers who will dispute a transaction once they’ve already received custom video clips, or photos. OnlyFans’ official policy on its website says the creator, not the company, foots the bill for a chargeback. (OnlyFans did not respond to requests for comment.)

Many models have taken to forums like Reddit to share their experiences, in which they say these alleged scammers will sometimes put in for a chargeback six months after receiving pictures or videos.

Transactions in cryptocurrencies are final, rendering chargebacks impossible.

[…]

UK-based escort agency VIP Passion started to accept bitcoin in 2013. Two years later, Backpage made a similar move into bitcoin, litecoin, and dogecoin after Visa and Mastercard refused to process payments for its “adult” section.

Visa said at the time that the company’s rules prohibited the network from “being used for illegal activity” and that Visa had a “long history of working with law enforcement to safeguard the integrity of the payment system.” Mastercard issued a similar statement, saying that the card company has rules prohibiting its cards from “being used for illegal or brand-damaging activities.”

[…]

Stabile warns there are still barriers to mass crypto adoption among sex workers.

For one, there’s a steep learning curve for both workers and customers. Sex workers have written and circulated guides online on how to use crypto, but a sizable knowledge gap remains.

It is also difficult to get some customers to spend their bitcoin on adult content.

“They generally use it as a store of value,” says Stabile. “It’s a speculative currency.”

Knox says often clients choose not to pay her in crypto.

“That’s the hurdle that we’re at right now. We can take it all day long, but until people start using it and start paying us with it, it’s not going to really take off for adoption,” said Knox.

Sex workers who do accept crypto also have to contend with volatile prices, which can cut into their earnings. For instance, bitcoin is down more than 40% from its November all-time high.

[…]

DiAngelo says that in the early days of crypto, she would use bitcoin ATMs at liquor stores and gas stations to deposit cash to buy bitcoin. These machines charge commissions above and beyond the cost of the transaction.

Another major problem relates to the rules that govern cryptocurrency exchanges. Many platforms like Coinbase require know-your-customer, or KYC compliance. In practice, that means having to connect an ID and bank account to the platform – a non-starter for many working in the industry.

Because of this, some workers later find they can’t cash out the crypto they have earned for products or services rendered.

[…]

“For people like me making millions of dollars, a thirty day notice from OnlyFans would be the end of us. Crypto really feels like it’s kinda it, otherwise we’re going to be controlled forever and who knows the kind of content they’re going to continue to ban. They can turn you off tomorrow.”

Source: Bitcoin a lifeline for sex workers, like ex-nurse making $1.3 million

Dutch watchdog fines Apple $5.7 million for 3rd weekly time in App Store dispute

The Dutch antitrust watchdog on Monday fined apple Apple (AAPL.O) 5 million euros ($5.72 million) for a third time for failing to allow software application makers in the Netherlands to use non-Apple payment methods for dating apps listed in the company’s App Store.

The Authority for Consumers and Markets (ACM) has been levying weekly fines of 5 million euros on Apple since the company missed a Jan. 15 deadline to make changes ordered by the watchdog.

Apple, which could not immediately be reached for comment, has twice published information on its own blog about changes it is making to comply with the Dutch order. However, the ACM said on Monday it was not receiving enough information from the U.S. company to assess whether Apple was actually complying.

“ACM is disappointed in Apple’s behaviour and actions,” it said in a statement. It noted that Dutch courts have upheld its decision, which found that Apple’s behaviour violated competition law.

[…]

Source: Dutch watchdog fines Apple $5.7 million again in App Store dispute

The Alternative to Web Scraping. The “lazy” programmer’s guide to… | by Doug Guthrie

One of the better sites for financial data is Yahoo Finance. This makes it a prime target for web scraping by finance enthusiasts. There are nearly daily questions on StackOverflow that reference some sort of data retrieval (oftentimes through web scraping) from Yahoo Finance.

Web Scraping Problem #1

trying to test a code that scrap from yahoo finance

I’m a python beginner but I like to learn the language by testing it and trying it. so there is a yahoo web scraper…

stackoverflow.com

The OP is trying to find the current price for a specific stock, Facebook. Their code is below:

And that code produced the following output:

the current price: 216.08

It’s a pretty simple problem with an also simple web scraping solution. However, it’s not lazy enough. Let’s look at the next one.

Web Scraping Problem #2

Web Scraping Yahoo Finance Statistics — Code Errors Out on Empty Fields

I found this useful code snippet: Web scraping of Yahoo Finance statistics using BS4 I have simplified the code as per…

stackoverflow.com

The OP is trying to extract data from the statistics tab, the stock’s enterprise value and the number of shares short. His problem actually revolves around retrieving nested dictionary values that may or may not be there, but he seems to have found a better solution as far as retrieving data.

Take a look at line 3: the OP was able to find the data he’s looking for inside a variable in the javascript:

root.App.main = { .... };

From there, the data is retrieved pretty simply by accessing the appropriate nested keys within the dictionary, data. But, as you may have guessed, there is a simpler, lazier solution.

Lazy Solution #1

Look at the URL on line 3

Output:

{
    'quoteSummary': {
        'error': None,
        'result': [{
            'price': {
                'averageDailyVolume10Day': {},
                'averageDailyVolume3Month': {},
                'circulatingSupply': {},
                'currency': 'USD',
                'currencySymbol': '$',
                'exchange': 'NMS',
                'exchangeDataDelayedBy': 0,
                'exchangeName': 'NasdaqGS',
                'fromCurrency': None,
                'lastMarket': None,
                'longName': 'Facebook, Inc.',
                'marketCap': {
                    'fmt': '698.42B',
                    'longFmt': '698,423,836,672.00',
                    'raw': 698423836672
                },
                'marketState': 'REGULAR',
                'maxAge': 1,
                'openInterest': {},
                'postMarketChange': {},
                'postMarketPrice': {},
                'preMarketChange': {
                    'fmt': '-0.90',
                    'raw': -0.899994
                },
                'preMarketChangePercent': {
                    'fmt': '-0.37%',
                    'raw': -0.00368096
                },
                'preMarketPrice': {
                    'fmt': '243.60',
                    'raw': 243.6
                },
                'preMarketSource': 'FREE_REALTIME',
                'preMarketTime': 1594387780,
                'priceHint': {
                    'fmt': '2',
                    'longFmt': '2',
                    'raw': 2
                },
                'quoteSourceName': 'Nasdaq Real Time '
                'Price',
                'quoteType': 'EQUITY',
                'regularMarketChange': {
                    'fmt': '0.30',
                    'raw': 0.30160522
                },
                'regularMarketChangePercent': {
                    'fmt': '0.12%',
                    'raw': 0.0012335592
                },
                'regularMarketDayHigh': {
                    'fmt': '245.49',
                    'raw': 245.49
                },
                'regularMarketDayLow': {
                    'fmt': '239.32',
                    'raw': 239.32
                },
                'regularMarketOpen': {
                    'fmt': '243.68',
                    'raw': 243.685
                },
                'regularMarketPreviousClose': {
                    'fmt': '244.50',
                    'raw': 244.5
                },
                'regularMarketPrice': {
                    'fmt': '244.80',
                    'raw': 244.8016
                },
                'regularMarketSource': 'FREE_REALTIME',
                'regularMarketTime': 1594410026,
                'regularMarketVolume': {
                    'fmt': '19.46M',
                    'longFmt': '19,456,621.00',
                    'raw': 19456621
                },
                'shortName': 'Facebook, Inc.',
                'strikePrice': {},
                'symbol': 'FB',
                'toCurrency': None,
                'underlyingSymbol': None,
                'volume24Hr': {},
                'volumeAllCurrencies': {}
            }
        }]
    }
}the current price: 241.63

Lazy Solution #2

Again, look at the URL on line 3

Output:

{
    'quoteSummary': {
        'result': [{
            'defaultKeyStatistics': {
                'maxAge': 1,
                'priceHint': {
                    'raw': 2,
                    'fmt': '2',
                    'longFmt': '2'
                },
                'enterpriseValue': {
                    'raw': 13677747200,
                    'fmt': '13.68B',
                    'longFmt': '13,677,747,200'
                },
                'forwardPE': {},
                'profitMargins': {
                    'raw': 0.07095,
                    'fmt': '7.10%'
                },
                'floatShares': {
                    'raw': 637754149,
                    'fmt': '637.75M',
                    'longFmt': '637,754,149'
                },
                'sharesOutstanding': {
                    'raw': 639003008,
                    'fmt': '639M',
                    'longFmt': '639,003,008'
                },
                'sharesShort': {},
                'sharesShortPriorMonth': {},
                'sharesShortPreviousMonthDate': {},
                'dateShortInterest': {},
                'sharesPercentSharesOut': {},
                'heldPercentInsiders': {
                    'raw': 0.0025499999,
                    'fmt': '0.25%'
                },
                'heldPercentInstitutions': {
                    'raw': 0.31033,
                    'fmt': '31.03%'
                },
                'shortRatio': {},
                'shortPercentOfFloat': {},
                'beta': {
                    'raw': 0.365116,
                    'fmt': '0.37'
                },
                'morningStarOverallRating': {},
                'morningStarRiskRating': {},
                'category': None,
                'bookValue': {
                    'raw': 12.551,
                    'fmt': '12.55'
                },
                'priceToBook': {
                    'raw': 1.3457094,
                    'fmt': '1.35'
                },
                'annualReportExpenseRatio': {},
                'ytdReturn': {},
                'beta3Year': {},
                'totalAssets': {},
                'yield': {},
                'fundFamily': None,
                'fundInceptionDate': {},
                'legalType': None,
                'threeYearAverageReturn': {},
                'fiveYearAverageReturn': {},
                'priceToSalesTrailing12Months': {},
                'lastFiscalYearEnd': {
                    'raw': 1561852800,
                    'fmt': '2019-06-30'
                },
                'nextFiscalYearEnd': {
                    'raw': 1625011200,
                    'fmt': '2021-06-30'
                },
                'mostRecentQuarter': {
                    'raw': 1577750400,
                    'fmt': '2019-12-31'
                },
                'earningsQuarterlyGrowth': {
                    'raw': 0.114,
                    'fmt': '11.40%'
                },
                'revenueQuarterlyGrowth': {},
                'netIncomeToCommon': {
                    'raw': 938000000,
                    'fmt': '938M',
                    'longFmt': '938,000,000'
                },
                'trailingEps': {
                    'raw': 1.434,
                    'fmt': '1.43'
                },
                'forwardEps': {},
                'pegRatio': {},
                'lastSplitFactor': None,
                'lastSplitDate': {},
                'enterpriseToRevenue': {
                    'raw': 1.035,
                    'fmt': '1.03'
                },
                'enterpriseToEbitda': {
                    'raw': 6.701,
                    'fmt': '6.70'
                },
                '52WeekChange': {
                    'raw': -0.17621362,
                    'fmt': '-17.62%'
                },
                'SandP52WeekChange': {
                    'raw': 0.045882702,
                    'fmt': '4.59%'
                },
                'lastDividendValue': {},
                'lastCapGain': {},
                'annualHoldingsTurnover': {}
            }
        }],
        'error': None
    }
}{'AGL.AX': {'Enterprise Value': '13.73B', 'Shares Short': 'N/A'}}

The lazy alternatives simply altered the request from utilizing the front-end URL to a somewhat unofficial API endpoint, which returns JSON data. It’s simpler and results in more data! What about speed though (pretty sure I promised simpler, more data, and a faster alternative)? Let’s check:

web scraping #1 min time is 0.5678426799999997
lazy #1 min time is 0.11238783999999953
web scraping #2 min time is 0.3731000199999997
lazy #2 min time is 0.0864451399999993

The lazy alternatives are 4x to 5x faster than their web scraping counterparts!

You might be thinking though, “That’s great, but where did you find those URLs?”.

The Lazy Process

Think about the two problems we walked through above: the OP’s we’re trying to retrieve the data after it had been loaded into the page. The lazier solutions went right to the source of the data and didn’t bother with the front-end page at all. This is an important distinction and, I think, a good approach whenever you’re trying to extract data from a website.

Step 1: Examine XHR Requests

An XHR (XMLHttpRequest) object is an API available to web browser scripting languages such as JavaScript. It is used to send HTTP or HTTPs requests to a web server and load the server response data back into the script. Basically, it allows the client to retrieve data from a URL without having to do a full page refresh.

I’ll be using Chrome for the following demonstrations, but other browsers will have similar functionality.

  • If you’d like to follow along, navigate to https://finance.yahoo.com/quote/AAPL?p=AAPL
  • Open Chrome’s developer console. To open the developer console in Google Chrome, open the Chrome Menu in the upper-right-hand corner of the browser window and select More Tools > Developer Tools. You can also use the shortcut Option + ⌘ + J (on macOS), or Shift + CTRL + J (on Windows/Linux).
  • Select the “Network” tab

  • Then filter the results by “XHR”

  • Your results will be similar but not the same. You should notice though that there are a few requests that contain “AAPL”. Let’s start by investigating those. Click on one of the links in the left-most column that contain the characters “AAPL”.

  • After selecting one of the links, you’ll see an additional screen that provides details into the request you selected. The first tab, Headers, provides details into the request made by the browser and the response from the server. Immediately, you should notice the Request URL in the Headers tab is very similar to what was provided in the lazy solutions above. Seems like we’re on the right track.
  • If you select the Preview tab, you’ll see the data returned from the server.

  • Perfect! It looks like we just found the URL to get OHLC data for Apple!

Step 2: Search

Now that we’ve found some of the XHR requests that are made via the browser, let’s search the javascript files to see if we can find any more information. The commonalities I’ve found with the URLs relevant to the XHR requests are “query1” and “query2”. In the top-right corner of the developer’s console, select the three vertical dots and then select “Search” in the dropdown.

Search for “query2” in the search bar:

Select the first option. An additional tab will pop-up containing where “query2” was found. You should notice something similar here as well:

It’s the same variable that web scraping solution #2 targeted to extract their data. The console should give you an option to “pretty-print” the variable. You can either select that option or copy and paste the entire line (line 11 above) into something like https://beautifier.io/ or if you use vscode, download the Beautify extension and it will do the same thing. Once it’s formatted appropriately, paste the entire code into a text editor or something similar and search for “query2” again. You should find one result inside something called “ServicePlugin”. That section contains the URLs that Yahoo Finance utilizes to populate data in their pages. The following is taken right out of that section:

"tachyon.quoteSummary": {"path": "\u002Fv10\u002Ffinance\u002FquoteSummary\u002F{symbol}","timeout": 6000,"query": ["lang", "region", "corsDomain", "crumb", "modules",     "formatted"],"responseField": "quoteSummary","get": {"formatted": true}},

This is the same URL that is utilized in the lazy solutions provided above.

TL;DR

  • While web scraping can be necessary because of how a website is structured, it’s worth the effort investigating to see if you can find the source of the data. The resulting code is simpler and more data is extracted faster.
  • Finding the source of a website’s data is often found by searching through XHR requests or by searching through the site’s javascript files utilizing your browser’s developer console.

More Information

  • What if you can’t find any XHR requests? Check out The Alternative to Web Scraping, Part II: The DRY approach to retrieving web data

The Alternative to Web Scraping, Part II

The DRY approach to retrieving web data

towardsdatascience.com

  • If you’re interested specifically in the Yahoo Finance aspect of this article, I’ve written a python package, yahooquery, that exposes most of those endpoints in a convenient interface. I’ve also written an introductory article that describes how to use the package as well as a comparison to a similar one.

The (Unofficial) Yahoo Finance API

A Python interface to endless amounts of data

towardsdatascience.com

  • Please feel free to reach out if you have any questions or comments

Source: The Alternative to Web Scraping. The “lazy” programmer’s guide to… | by Doug Guthrie | Towards Data Science

Researchers create strong synthetic enamel similar to natural tooth covering

A team of researchers from Beihang University, the Peking University School and Hospital of Stomatology and the Michigan Institute of Translational Nanotechnology has developed a synthetic enamel with properties similar to natural tooth enamel. In their paper published in the journal Science, the group describes their enamel and how well it compared to natural enamel when tested.

[…]

Prior research has shown that the reason that human enamel is so strong and yet also slightly elastic is because it consists of tiny rods made of calcium that are packed tightly together like pencils in a box. In their new effort, the researchers attempted to mimic as closely as possible by producing a material using AIP-coated hydroxyapatite nanowires that were aligned in parallel using a freezing technique that involved applying polyvinyl alcohol.

The researchers applied the enamel to a variety of shapes, including human teeth, and then tested how well it performed. They found it had a high degree of stiffness, was strong and was also slightly elastic. They also found that on most of their tests, the synthetic enamel outperformed natural enamel.

The researchers plan to keep testing their material to make sure it will hold up under such as those found in the human mouth. They will also have to show that it is safe for use in humans and that it can be mass produced. They note that if their passes all such tests, it could be used in more than just dentistry—they suggest it could be used to coat pacemakers, for example, or to shore up bones that have been damaged or that have eroded due to use or disease.

Source: Researchers create strong synthetic enamel similar to natural tooth covering

Thousands of Planes Are Flying Empty and No One Can Stop Them

In December 2021, 27,591 aircraft took off or landed at Frankfurt airport—890 every day. But this winter, many of them weren’t carrying any passengers at all. Lufthansa, Germany’s national airline, which is based in Frankfurt, has admitted to running 21,000 empty flights this winter, using its own planes and those of its Belgian subsidiary, Brussels Airlines, in an attempt to keep hold of airport slots.

Although anti-air travel campaigners believe ghost flights are a widespread issue that airlines don’t publicly disclose, Lufthansa is so far the only airline to go public about its own figures. In January, climate activist Greta Thunberg tweeted her disbelief over the scale of the issue. Unusually, she was joined by voices within the industry. One of them was Lufthansa’s own chief executive, Carsten Spohr, who said the journeys were “empty, unnecessary flights just to secure our landing and takeoff rights.” But the company argues that it can’t change its approach: Those ghost flights are happening because airlines are required to conduct a certain proportion of their planned flights in order to keep slots at high-trafficked airports.

A Greenpeace analysis indicates that if Lufthansa’s practice of operating no-passenger flights were replicated equally across the European aviation sector, it would mean that more than 100,000 “ghost flights” were operating in Europe this year, spitting out carbon dioxide emissions equivalent to 1.4 million gas-guzzling cars. “We’re in a climate crisis, and the transport sector has the fastest-growing emissions in the EU,” says Greenpeace spokesperson Herwig Schuster. “Pointless, polluting ‘ghost flights’ are just the tip of the iceberg.”

Aviation analysts are split on the scale of the ghost flight problem. Some believe the issue has been overhyped and is likely not more prevalent than the few airlines that have admitted to operating them. Others say there are likely tens of thousands of such flights operating—with their carriers declining to say anything because of the PR blowback.

“The only reason we have [airport] slots is that it recognizes a shortage of capacity at an airport,” says John Strickland of JLS Consulting, an aviation consultant. “If there wasn’t any shortage of capacity, airlines could land and take off within reason whenever they want to.” However, a disparity between the volume of demand for takeoff and landing slots and the number of slots available at key airports means that airlines compete fiercely for spaces. In 2020, 62 million flights took place at the world’s airports, according to industry body Airports Council International. While that number sounds enormous, it’s down nearly 40 percent year on year. To handle demand, more than 200 airports worldwide operate some kind of slot system, handling a combined 1.5 billion passengers. If you board a flight anywhere in the world, there’s a 43 percent chance your flight is slot managed.

Airlines even pay their competitors to take over slots: Two highly prized slots at London Heathrow airport reportedly changed hands for $75 million in 2016, when tiny cash-rich airline Oman Air made Air France-KLM an offer it couldn’t refuse for a sleepy 5.30 am arrival from Muscat to the UK capital.

[…]

Source: Thousands of Planes Are Flying Empty and No One Can Stop Them | WIRED

Developers react to Apples 27% commission with astonishment, anger

Developers reacted with astonishment and anger at Apple’s 27% commission policy as a minimal form of compliance with a new antitrust law regarding the App Store.

One leading developer described the move as ‘vile,’ while another said Apple is deliberately ensuring it would cost developers more to opt-out of Apple’s payment system than it would to remain within it …

 

Background

Dutch regulators, like those in South Korea, ordered that Apple allow developers to opt-out of the App Store payment platform. Apple initially said that it would comply, but didn’t give any details.

The company today announced that it would reduce its commission by only three percent for those who chose to do so, and would also impose onerous administrative overheads – such as applying for permission to use a specific API, maintaining a separate version of the app, and filing reports with Apple.

[…]

Marco Arment highlighted the conditions imposed by Apple:

  • Separate app, only available in Netherlands
  • Cannot also support IAP
  • Must display scary sheets before payment
  • Website links are all to a single URL specified in Info.plist with no parameters
  • Must submit monthly report to Apple listing EVERY external transaction

Adding:

And after you pay your ~3% to your payment processor, Apple’s 27% commission takes you right back up to 30%. Glorious. Come on, THIS is comedy. Amazing, ridiculous comedy. I’d be surprised if a single app ever took them up on this. (And that’s exactly by design.)

[…]

Source: Developers react to 27% commission with astonishment, anger – 9to5Mac

Suicide Hotline Collected, Monetized The Data Of Desperate People, Because Of Course It Did

Crisis Text Line, one of the nation’s largest nonprofit support options for the suicidal, is in some hot water. A Politico report last week highlighted how the company has been caught collecting and monetizing the data of callers… to create and market customer service software. More specifically, Crisis Text Line says it “anonymizes” some user and interaction data (ranging from the frequency certain words are used, to the type of distress users are experiencing) and sells it to a for-profit partner named Loris.ai. Crisis Text Line has a minority stake in Loris.ai, and gets a cut of their revenues in exchange.

As we’ve seen in countless privacy scandals before this one, the idea that this data is “anonymized” is once again held up as some kind of get out of jail free card:

“Crisis Text Line says any data it shares with that company, Loris.ai, has been wholly “anonymized,” stripped of any details that could be used to identify people who contacted the helpline in distress. Both entities say their goal is to improve the world — in Loris’ case, by making “customer support more human, empathetic, and scalable.”

But as we’ve noted more times than I can count, “anonymized” is effectively a meaningless term in the privacy realm. Study after study after study has shown that it’s relatively trivial to identify a user’s “anonymized” footprint when that data is combined with a variety of other datasets. For a long time the press couldn’t be bothered to point this out, something that’s thankfully starting to change.

[…]

Source: Suicide Hotline Collected, Monetized The Data Of Desperate People, Because Of Course It Did | Techdirt

North Korea Hacked Him. So One Guy Took Down Its Internet

For the past two weeks, observers of North Korea’s strange and tightly restricted corner of the internet began to notice that the country seemed to be dealing with some serious connectivity problems. On several different days, practically all of its websites—the notoriously isolated nation only has a few dozen—intermittently dropped offline en masse, from the booking site for its Air Koryo airline to Naenara, a page that serves as the official portal for dictator Kim Jong-un’s government. At least one of the central routers that allow access to the country’s networks appeared at one point to be paralyzed, crippling the Hermit Kingdom’s digital connections to the outside world.

[…]

But responsibility for North Korea’s ongoing internet outages doesn’t lie with US Cyber Command or any other state-sponsored hacking agency. In fact, it was the work of one American man in a T-shirt, pajama pants, and slippers, sitting in his living room night after night, watching Alien movies and eating spicy corn snacks—and periodically walking over to his home office to check on the progress of the programs he was running to disrupt the internet of an entire country.

Just over a year ago, an independent hacker who goes by the handle P4x was himself hacked by North Korean spies. P4x was just one victim of a hacking campaign that targeted Western security researchers with the apparent aim of stealing their hacking tools and details about software vulnerabilities. He says he managed to prevent those hackers from swiping anything of value from him. But he nonetheless felt deeply unnerved by state-sponsored hackers targeting him personally—and by the lack of any visible response from the US government.

So after a year of letting his resentment simmer, P4x has taken matters into his own hands. “It felt like the right thing to do here. If they don’t see we have teeth, it’s just going to keep coming,” says the hacker. (P4x spoke to WIRED and shared screen recordings to verify his responsibility for the attacks but declined to use his real name for fear of prosecution or retaliation.)

[…]

P4x says he’s found numerous known but unpatched vulnerabilities in North Korean systems that have allowed him to singlehandedly launch “denial-of-service” attacks on the servers and routers the country’s few internet-connected networks depend on.

[…]

he named, as an example, a known bug in the web server software NginX that mishandles certain HTTP headers, allowing the servers that run the software to be overwhelmed and knocked offline. He also alluded to finding “ancient” versions of the web server software Apache,

[…]

“It’s pretty interesting how easy it was to actually have some effect in there.”

[…]

He acknowledges that his attacks amount to no more than “tearing down government banners or defacing buildings,” as he puts it. But he also says that his hacking has so far focused on testing and probing to find vulnerabilities. He now intends to try actually hacking into North Korean systems, he says, to steal information and share it with experts. At the same time, he’s hoping to recruit more hacktivists to his cause with a dark website he launched Monday called the FUNK Project—i.e. “FU North Korea”—in the hopes of generating more collective firepower.

[…]

he was nonetheless shocked and appalled by the realization that he’d been personally targeted by North Korea.

P4x says he was later contacted by the FBI but was never offered any real help to assess the damage from North Korea’s hacking or to protect himself in the future. Nor did he ever hear of any consequences for the hackers who targeted him, an open investigation into them, or even a formal recognition from a US agency that North Korea was responsible. It began to feel, as he put it, like “there’s really nobody on our side.”

[…]

While he acknowledges that his attacks likely violate US computer fraud and hacking laws, he argues he hasn’t done anything ethically wrong. “My conscience is clear,” he says.

[…]

Source: North Korea Hacked Him. So He Took Down Its Internet | WIRED

Regulators find Europe’s ad-tech industry acted unlawfully, violates GDPR

After a years-long process, data protection officials across the European Union have ruled that Europe’s ad tech industry has been operating unlawfully. The decision, handed down by Belgium’s APD (.PDF) and agreed by regulators across the EU, found that the system underpinning the industry violated a number of principles of the General Data Protection Regulations (GDPR). The Irish Council for Civil Liberties has declared victory in its protracted battle against the authority which administers much of the advertising industry on the continent: IAB Europe.

At the heart of this story is the use of the Transparency and Consent Framework (TCF), a standardized process to enable publishers to sell ad-space on their websites. This framework, set by IAB Europe, is meant to provide legal cover — in the form of those consent pop-ups which blight websites — enabling a silent, digital auction system known-as Real-Time Bidding (RTB). But both the nature of the consent given when you click a pop-up, and the data collected as part of the RTB process have now been deemed to violate the GDPR, which governs privacy rights in the bloc.

Back in December, I wrote a deep (deep) dive on this situation*, and the potential privacy violations that the RTB process caused

[…]

The APD has ruled that any and all data collected as part of this Real-Time Bidding process must now be deleted. T

[…]

Regulators have also handed down an initial fine of €250,000 to IAB Europe and ordered the body to effectively rebuild the ad-tech framework it currently uses. This includes making the system GDPR compliant (if such a thing is possible) and appoint a dedicated Data Protection Officer.

[…]

 

Source: Regulators find Europe’s ad-tech industry acted unlawfully | Engadget

Blockchain platform Wormhole says it’s retrieved the $324M stolen by hackers

[…]

Hackers stole more than $324 million in cryptocurrency from Wormhole, the developers behind the popular blockchain bridge confirmed Wednesday.

The platform provides a connection that allows for the transfer of cryptocurrency between different decentralized-finance blockchain networks. Wormhole said in a series of tweets Wednesday afternoon that thieves made off with 120,000 wETH, or wrapped ethereum, worth nearly $324 million at current exchange rates. The platform’s network was also taken offline for maintenance.

[…]

Wormhole on Thursday confirmed via Twitter that “all funds have been restored” and its services are back up. It also promised to share a full incident report.

Source: Blockchain platform Wormhole says it’s retrieved the $324M stolen by hackers – CNET

Google adds new opt out tracking for Workspace Customers

[…]

according to a new FAQ posted on Google’s Workplace administrator forum. At the end of that month, the company will be adding a new feature—“Workspace search history”—that can continue to track these customers, even if they, or their admins, turn activity tracking off.

The worst part? Unlike Google’s activity trackers that are politely defaulted to “off” for all users, this new Workplace-specific feature will be defaulted to “on,” across Workspace apps like Gmail, Google Drive, Google Meet, and more.

[…]

Luckily, they can turn this option off if they want to, the same way they could turn off activity settings until now. According to Google, the option to do so will be right on the “My Activity” page once the feature goes live, right alongside the current options to flip off Google’s ability to keep tabs on their web activity, location history, and YouTube history. On this page, Google says the option to turn off Workspace history will be located on the far lefthand side, under the “Other Google Activity” tab.

[…]

Source: Google Makes Opting Out Harder for Workspace Customers

Intel’s $1.2bn EU antitrust fine cancelled by court 12 years after Intel didn’t pay up

Intel Corporation no longer has to pay a €1.06bn ($1.2bn, £890m) fine imposed by the European Commission (EC) in 2009 for abusing its dominance of the chip market.

On Wednesday, the General Court of the European Union annulled the EC antitrust penalty [PDF] after previously upholding it in 2014 [PDF].

After rival AMD complained in 2000 and again in 2003 that Intel was engaging in anti-competitive conduct by offering its hardware partners rebates for using Intel’s x86 chips, an EC antitrust investigation that got underway in 2004 and concluded in 2009 with a €1.06 billion penalty against Chipzilla.

The EC at the time found Intel’s conduct between October 2002 and December 2007 to be anti-competitive.

“The evidence gathered by the Commission led to the conclusion that Intel’s conditional rebates and payments induced the loyalty of key OEMs and of a major retailer, the effects of which were complementary in that they significantly diminished competitors’ ability to compete on the merits of their x86 CPUs,” the EC said in its 2009 decision. “Intel’s anti-competitive conduct thereby resulted in a reduction of consumer choice and in lower incentives to innovate.”

[…]

The ruling suggests that EU trustbusters won’t be able to constrain corporate behavior if alleged misconduct fails to fit within the limited definition of competitive abuse under EU law (Article 102 TFEU). According to the Associated Press, EC VP Margrethe Vestager said at a press briefing in Brussels that the EC needs more time to consider what comes next.

[…]

Source: Intel’s $1.2bn EU antitrust fine cancelled by court • The Register

Which begs the question – why is China leading the way in anti-competitive lawmaking?

LG Announces New Ad Targeting Features for TVs – wait, wtf, I bought my TV, not a service!

[… ]

there are plenty of cases where you throw down hundreds of dollars for a piece of hardware and then you end up being the product anyway. Case in point: TVs.

On Wednesday, the television giant LG announced a new offering to advertisers that promises to be able to reach the company’s millions of connected devices in households across the country, pummeling TV viewers with—you guessed it—targeted ads. While ads playing on your connected TV might not be anything new, some of the metrics the company plans to hand over to advertisers include targeting viewers by specific demographics, for example, or being able to tie a TV ad view to someone’s in-store purchase down the line.

If you swap out a TV screen for a computer screen, the kind of microtargeting that LG’s offering doesn’t sound any different than what a company like Facebook or Google would offer. That’s kind of the point.

[…]

Aside from being an eyesore that literally no TV user wants, these ads come bundled with their own privacy issues, too. While the kinds of invasive tracking and targeting that regularly happens with the ads on your Facebook feed or Google search results are built off of more than a decade’s worth of infrastructure, those in the connected television (or so-called “CTV”) space are clearly catching up, and catching up fast. Aside from what LG’s offering, there are other players in adtech right now that offer ways to connect your in-app activity to what you watch on TV, or the billboards you walk by with what you watch on TV. For whatever reason, this sort of tech largely sidesteps the kinds of privacy snafus that regulators are trying to wrap their heads around right now—regulations like CPRA and GDPR are largely designed to handle your data is handled on the web, not on TV.

[…]

The good news is that you have some sort of refuge from this ad-ridden hell, though it does take a few extra steps. If you own a smart TV, you can simply not connect it to the internet and use another device—an ad-free set-top box like an Apple TV, for instance—to access apps. Sure, a smart TV is dead simple to use, but the privacy trade-offs might wind up being too great.

Source: LG Announces New Ad Targeting Features for TVs

More Than 80% of NFTs Created for Free on OpenSea Are Fraud or Spam, Company Says

[…]

OpenSea has revealed just how much of the NFT activity on its platform is defined by fakery and theft, and it’s a lot. In fact, according to the company, nearly all of the NFTs created for free on its platform are either spam or plagiarized.

The revelation began with some drama. On Thursday, popular NFT marketplace OpenSea announced that it would limit how many times a user could create (or “mint”) an NFT for free on the platform using its tools to 50. So-called “lazy minting” on the site lets users skip paying a blockchain gas fee when they create an NFT on OpenSea (with the buyer eventually paying the fee at the time of sale), so it’s a popular option especially for people who don’t have deep pockets to jumpstart their digital art empire.

This decision set off a firestorm, with some projects complaining that this was an out-of-the-blue roadblock for them as they still needed to mint NFTs but suddenly couldn’t. Shortly after, OpenSea reversed course and announced that it would remove the limit, as well as provided some reasoning for the limit in the first place: The free minting tool is being used almost exclusively for the purposes of fraud or spam.

[…]

Source: More Than 80% of NFTs Created for Free on OpenSea Are Fraud or Spam, Company Says

Finnish diplomats were targeted by NSO Pegasus spyware

Finland’s government says the mobile devices of its diplomats have been hacked using Pegasus spyware.

The Finnish foreign ministry stated on Friday that some of its officials abroad had been targeted by the sophisticated software.

“The highly sophisticated malware has infected users’ Apple or Android telephones without their noticing and without any action from the user’s part,” the Foreign Ministry said in a statement.

“Through the spyware, the perpetrators may have been able to harvest data from the device and exploit its features.”

[…]

NSO says it only sells Pegasus to governments for the purpose of fighting crime and terrorism.

But an investigation last year revealed that the spyware had been used to target journalists, activists and politicians in a number of countries — including France, Spain, and Hungary.

A recent Citizen Lab report also found that critics of Poland’s right-wing government were hacked using Pegasus.

[…]

Source: Finnish diplomats were targeted by Pegasus spyware, says foreign ministry | Euronews

A Chinese Satellite Just Grappled Another And Pulled It Out Of Orbit

Chinese satellite was observed grabbing another satellite and pulling it out of its normal geosynchronous orbit and into a “super-graveyard drift orbit.” The maneuver raises questions about the potential applications of these types of satellites designed to maneuver close to other satellites for inspection or manipulation and adds to growing concerns about China’s space program overall.

On January 22, China’s Shijian-21 satellite, or SJ-21, disappeared from its regular position in orbit during daylight hours when observations were difficult to make with optical telescopes. SJ-21 was then observed executing a “large maneuver” to bring it closely alongside another satellite, a dead BeiDou Navigation System satellite. SJ-21 then pulled the dead satellite out of its normal geosynchronous orbit and placed it a few hundred miles away in what is known as a graveyard orbit. These distant orbits are designated for defunct satellites at the end of their lives and are intended to reduce the risk of collision with operational assets.

The unusual maneuver was observed by telescopes belonging to commercial space awareness firm Exoanalytic Solutions. During a webinar hosted by the Center for Strategic and International Studies (CSIS) this week, Exoanalytic Solutions’ Brien Flewelling said the SJ-21 satellite “appears to be functioning as a space tug.” Space Command did not respond to a request for comment, Breaking Defense reports.

Space Force has been increasingly turning to commercial space companies to provide a variety of data and services to boost its situational awareness, and to that end, Joint Task Force-Space Defense awarded Exoanalytic Solutions a contract in 2021 to provide space domain data. “Comms, data relay, remote sensing, and even ISR and some other things — [these] capabilities are increasingly available in the commercial market,” Space Force deputy Lt. Gen. David Thompson said last year.

SJ-21, or Shijian-21, was launched in October 2021 atop a Long March-3B rocket. The satellite is officially designated as an On-Orbit Servicing, Assembly, and Manufacturing, or OSAM satellite, a broad class of satellites designed with capabilities to get close to and interact with other satellites. Such systems could enable a wide range of applications including extending the life of existing satellites, assembling satellites in orbit, or performing other maintenance and repairs. According to Chinese state news outlets, SJ-21 was designed to “test and verify space debris mitigation technologies.”

[…]

Source: A Chinese Satellite Just Grappled Another And Pulled It Out Of Orbit

How normal am I? – Let an AI judge you

This is an art project by Tijmen Schep that shows how face detection algoritms are increasingly used to judge you. It was made as part of the European Union’s Sherpa research program.

No personal data is sent to our server in any way. Nothing. Zilch. Nada. All the face detection algorithms will run on your own computer, in the browser.

In this ‘test’ your face is compared with that of all the other people who came before you. At the end of the show you can, if you want to, share some anonimized data. That will then be used to re-calculate the new average. That anonymous data is not shared any further.

Source: How normal am I?

Stackable artificial leaf uses less power than lightbulb to capture 100 times more carbon than other systems

Engineers at the University of Illinois Chicago have built a cost-effective artificial leaf that can capture carbon dioxide at rates 100 times better than current systems. Unlike other carbon capture systems, which work in labs with pure carbon dioxide from pressurized tanks, this artificial leaf works in the real world. It captures carbon dioxide from more diluted sources, like air and flue gas produced by coal-fired power plants, and releases it for use as fuel and other materials.

[..]

Illustration of a carbon capture process designed by UIC College of Engineering scientists. Carbon dioxide from air or flue gas is absorbed by a dry organic solution to form bicarbonate ions, which migrate across a membrane and are dissolved in a liquid solution to concentrated CO2. Carbon atoms are shown in red, oxygen atoms are shown in blue and hydrogen atoms are shown in white. (Credit: Aditya Prajapati/UIC)

Using a previously reported theoretical concept, the scientists modified a standard artificial leaf system with inexpensive materials to include a water gradient — a dry side and a wet side — across an electrically charged membrane.

On the dry side, an organic solvent attaches to available carbon dioxide to produce a concentration of bicarbonate, or baking soda, on the membrane. As bicarbonate builds, these negatively charged ions are pulled across the membrane toward a positively charged electrode in a water-based solution on the membrane’s wet side. The liquid solution dissolves the bicarbonate back into carbon dioxide, so it can be released and harnessed for fuel or other uses.

The electrical charge is used to speed up the transfer of bicarbonate across the membrane.

When they tested the system, which is small enough to fit in a backpack, the UIC scientists found that it had a very high flux — a rate of carbon capture compared with the surface area required for the reactions — of 3.3 millimoles per hour per 4 square centimeters. This is more than 100 times better than other systems, even though only a moderate amount of electricity (0.4 KJ/hour) was needed to power the reaction, less than the amount of energy needed for a 1 watt LED lightbulb. They calculated the cost at $145 per ton of carbon dioxide, which is in line with recommendations from the Department of Energy that cost should not exceed around $200 per ton.

[…]

The UIC scientists report on the design of their artificial leaf and the results of their experiments in “Migration-assisted, moisture gradient process for ultrafast, continuous CO2 capture from dilute sources at ambient conditions,” which is published in Energy & Environmental Science.

[…]

Source: Stackable artificial leaf uses less power than lightbulb to capture 100 times more carbon than other systems | UIC Today

polkit has been allowing root for 12+ years

[…]Polkit, previously known as PolicyKit, is a tool for setting up policies governing how unprivileged processes interact with privileged ones. The vulnerability resides within polkit’s pkexec, a SUID-root program that’s installed by default on all major Linux distributions. Designated CVE-2021-4034, the vulnerability has been given a CVSS score of 7.8.

Bharat Jogi, director of vulnerability and threat research at Qualys, explained in a blog post that the pkexec flaw opens the door to root privileges for an attacker. Qualys researchers, he said, have demonstrated exploitation on default installations of Ubuntu, Debian, Fedora, and CentOS, and other Linux distributions are presumed to be vulnerable as well.

“This vulnerability has been hiding in plain sight for 12+ years and affects all versions of pkexec since its first version in May 2009,” said Jogi, pointing to commit c8c3d83, which added a pkexec command.

The problem occurs when pkexec‘s main() function processes command-line arguments and argc – the ARGument Count – is zero. The function tries to access the list of arguments anyway, and ends up trying to use an empty argv – the ARGument Vector of command-line argument strings. As a result, out-of-bounds memory gets read and written, which an attacker can exploit to inject an environment variable that can cause arbitrary code to be loaded from storage and run by the program as root.

[…]

At least the exploitation technique proposed by Qualys – injecting the GCONV_PATH variable into pkexec‘s environment to execute a shared library as root – leaves traces in log files.

[…]

Source: Linux system service polkit has make-me-root security flaw • The Register

Google Drive flags single-digit files over copyright

A funny thing happened on Google Drive overnight. Seemingly innocuous files started being flagged as violating the search behemoth’s terms of service over copyright infringement.

Dr Emily Dolson, assistant professor at Michigan State University, was one of those affected after she attempted to upload a file containing a single digit, “1”.

There wasn’t a lot of detail in the warning, only that Googles Drive’s Copyright Infringement policy had been violated and that no review could be requested for the restriction, both of which are a bit worrying for people concerned about the dead hand of AI being used as arbiter in such matters.

What had upset Google? The digit or the output04.txt filename? Certainly the number “1” does turn up in all manner of copyrighted works, although we don’t think anyone’s tried to trademark the character. Most recently, Snap made a spectacle of itself by trying to trademark the word “Spectacles”.

Could Google be trying to up the ante, and is it aware that Microsoft has its own cloud storage named OneDrive? Redmond already had to ditch SkyDrive after a well-known broadcaster took exception to it. We can’t imagine Nadella and co liking the sound of “Number Two Drive” for a variety of reasons.

More likely, the issue was more of a screw-up than conspiracy with both Google staffers and the Google Drive social media mouthpiece responding to confirm that the team was aware of the issue and working on it.

Additional users reported problems with other numbers, including “0”, while wags over on Hacker News pointed to the relevant Onion article.

Because there’s always an Onion article where automation drives swathes of the IT world beyond satire.

Things seem OK now (at least as far as our testing is concerned), although we have asked Google to explain itself. We will update this piece if it does so.

Whatever the fix was, we suspect it wasn’t this. ®

Source: Google Drive flags single-digit files over copyright

Flying car wins airworthiness certification – BBC News

A flying car capable of hitting speeds over 100mph (160kmh) and altitudes above 8,000ft (2,500m) has been issued with a certificate of airworthiness by the Slovak Transport Authority.

The hybrid car-aircraft, AirCar, is equipped with a BMW engine and runs on regular petrol-pump fuel.

It takes two minutes and 15 seconds to transform from car into aircraft.

The certification followed 70 hours of flight testing and more than 200 take-offs and landings, the company said.

Source: Flying car wins airworthiness certification – BBC News

How to Download Everything Amazon Knows About You (It’s a Lot)

[…]To be clear, data collection is far from an Amazon-specific problem; it’s pretty much par for the course when it comes to tech companies. Even Apple, a company vocal about user privacy, has faced criticism in the past for recording Siri interactions and sharing them with third-party contractors.

The issue with Amazon, however, is the extent to which they collect and archive your data. Just about everything you do on, with, and around an Amazon product or service is logged and recorded. Sure, you might not be surprised to learn that when you visit Amazon’s website, the company logs your browsing history and shopping data. But it goes far beyond that. Since Amazon owns Whole Foods, it also saves your shopping history there. When you watch video content through its platforms, it records all of that information, too.

Things get even creepier with other Amazon products. If you read books on a Kindle, Amazon records your reading activity, including the speed of your page turns (I wonder if Bezos prefers a slow or fast page flip); if you peered into your Amazon data, you might find something similar to what a Reuter’s reporter found: On Aug. 8 2020, someone on that account read The Mitchell Sisters: A Complete Romance Series from 4:52 p.m. through 7:36 p.m., completing 428 pages. (Nice sprint.)

If you have one of Amazon’s smart speakers, you’re on the record with everything you’ve ever uttered to the device: When you ask Alexa a question or give it a command, Amazon saves the audio files for the entire interaction. If you know how to access you data, you can listen to every one of those audio files, and relive moments you may or may not have realized were recorded.

Another Reuters reporter found Amazon saved over 90,000 recordings over a three-and-a-half-year period, which included the reporter’s children asking Alexa questions, recordings of those same children apologizing to their parents, and, in some cases, extended conversations that were outside the scope of a reasonable Alexa query.

Unfortunately, while you can access this data, Amazon doesn’t make it possible to delete much of it. You can tweak your privacy settings you stop your devices from recording quite as much information. However, once logged, the main strategy to delete it is to delete the entire account it is associated with. But even if you can’t delete the data while sticking with your account, you do have a right to see what data Amazon has on you, and it’s simple to request.

How to download all of your Amazon data

To start, , or go to Amazon’s Help page. You’ll find the link under Security and Privacy > More in Security & Privacy > Privacy > How Do I Request My Data? Once there, click the “Request My Data” link.

From the dropdown menu, choose the data you want from Amazon. If you want everything, choose “Request All Your Data.” Hit “Submit Request,” then click the validation link in your email. That’s it. Amazon makes it easy to see what the have on you, probably because they know you can’t do anything about it.

[Reuters]

Source: How to Download Everything Amazon Knows About You (It’s a Lot)