Ethereum-based DeFi apps have accumulated a staggering $50bn in total value locked (TVL), but many retail investors are priced out by high transaction costs (“gas fees”). New alternatives to Ethereum appear every day, but in order to maximise returns, an investor needs a variety of components to all to come together in a stack.
The blockchain itself provides the basis infrastructure for the decentralised apps (dApps) on top. The dominant blockchain is Ethereum, with its native Ether token (ETH). Any user wanting to invest in the Ethereum stack will need Ether to pay for transactions (“gas”).
All alternatives discussed in…
I was working with a client on analysing Athena query logs. We wanted to partition the log data so that we don’t scan the entire log set with Athena every day.
It turns out this was not as easy as you may think. This is due to the fact the the CloudTrail logs are not partitioned in a Hive way.
The root directory of my cloud trail logs is:
Underneath this root directory, there are subdirectories for each region.
Underneath this, one for each year, thereunder for the month, and thereunder for the day.
So, for example: s3://aws-cloudtrail-logs-255149284406–4cec155e/AWSLogs/255149284406/CloudTrail/us-west-2/2020/12/03/ contains…
This post discusses data engineering with a concrete example. The dataset was produced by Antonis Papasavva et al. A pyspark notebook is provided.
The dataset is about 27GB in zstd-compressed format. I followed the instructions provided above and copied the uncompressed file to S3.
Next, I wanted to convert the dataset to a format suitable for analysis with popular cloud data analysis tools. I chose to focus on Spark and Athena.
Spark and Athena have many similarities: they both can handle large data sets, use more or less the same file formats (e.g. …
This must have been the most important week in Crypto and DeFi in a while. The day after $YAM, $CRV was given birth by a “community member”.
I used a variety of tools to understand what happened, and I invite you to follow along.
This is the rough outline of the events.
Airlines decide about their schedules twice a year, for their summer (roughly 1 April —31 October) and winter (1 November — 31 March) schedules. Summer and winter (based on the Northern hemisphere) reflect different demand patterns. Nothing stops airlines from having seasonal flights (maybe July-August only) for other marketing reasons.
They submit their schedules to several places, including:
Rather than saying: flight AB 01 from LHR to JFK is daily 0700–1000, they actually submit every single flight that is supposed…