@alexcleac @alexcleac@mastodon.technology

The thing you have to remember about #pyspark when using mapPartitions: first argument is *always* a generator. Make sure you don't exhaust it before actually using in your python code.

#Spark #Python

Never use thread local database connection while trying to write something from #spark distributed way in #python!

#TIL in #python

<code lang="python">
from unittest.mock import Mock

> schema = {'a': 1}
... mock = Mock(**schema)
... mock.a
1
</code>

#TIL #python groupby function groups by in local context. It actually groups only things that belong to single group inside continuous group.

Example:
group by of this stroke:
[k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B

What I actually expected it to do:

[k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D

Also, I am very not happy with #python api for #spark. The #scala API is way more powerful for making complicated things, while python one feels like a tool for making aggregation charts

I am playing around and implementing some #monad types in #python

https://gist.github.com/cleac/f30d99ac9df31a8383766e9b2eaf97d8

Here is what I've got. Pretty simple and not fully implemented, but totally fun :)

Now I can tell, why I think that #python and #javascript are not that good for starters. You get used to with the fact that all your applications will execute in single thread and there is no concurrency in your code at all. But when you get to paralel processing of data, you make stupid errors of context syncrhonization.

Back to the social media.

Was debugging #Spark again for several previous days. Things that I've learnt during another deep debugging session:

1. It seems that it doesn't like when you try reading old checkpoints
2. Do not use different schemass for serialization in same application, or you will spend a lot of time debugging them.
3. #Scala is statically typed language that allows you to make things you can do in #Python, but with guarantee that if it compiles then it will at least work.

People at work start considering me as “oh, you’re that guy who post much of things about Scala, why do you do it?”. And I respond that I do #Scala. Then they ask me about y opinion on it. And I tell, “We’ll it’s more difficult than #python” 😂😂

Giving my laptop some hard time in recoding thing from mp3 to ogg. Also, first time in my life I found out that it is easier to write a #python script that to write #bash script to work with files with spaces. It took for about several hours to try making script in bash and about 15 minutes to write a python script that does literally the same, faster and more stable! #Linux #nerd #life is hard :D

Last evening I was doing two things at the same time:

1 - I was writing a universal DSL for querying languages using native #Python type hinting (I call it naiiveORM). Will open source it when ready.

2 - I was playing Serious Sam: First Encounter. I’ve played it a bit when I was a child, but now I like it so much. And all that on #Linux thanks to #Proton.

More I dive into #Scala more I understand that there is something wrong with #Python.

Reason is very simple: Python declares to be expressive and easily language, but lately I feel that most of the time I just fight the interpreter to express my thoughts. And with Scala, I just write my thoughts straightforward, and they just work. Easily and fast, without any hacks and fighting language

It seems that #Scala developer had bitten me. I am spending my morning covering module with static types in #Python just because it fails and I want to prevent further bugs in implementation 😂

#Python development feels pretty decent in #vim if you use #LSP. It is one of the best things made recently, nevertheless it was created by Microsoft.

Why? Because it makes development tools agnostic to editor you use. You use the very same tools, when you use Atom, ViM and Emacs or anything else that supports those protocols.

This is what should be made years ago, when the "editors war" just started.

@sir can you give some insights on what is sr.ht built on? Was just curions, I saw that meta.sr.ht is built in #python (I am amazed with that), what are other parts built on?

By the way, I am very happy to join sr.ht 😄

#mypy #python

Teacher:
Let's look at this code:
`comment: str = None`
Is the type correct?
mypy:
Yes, everything is just fine!

Found out how to make distribution package in #python. It was pretty, and I make miniflux-cli lighter on cpu and without re-render blinking while terminal is idling. Kind of version 0.0.2, proof of concept
https://github.com/cleac/miniflux-cli

The more I explore #curses module #python #docs, the more I gain feeling that I write abstraction over abstraction that already has this things implemented. And manual doesn't help in any way!

Writing cli applications in #Python is a bit painful, because damn, you have to invent a wheel, and this all trying to make it still beautiful and shiny. Maybe I'm doing something wrong probably :D

Found out how powerfull is #trafaret library for #python. I used to think that it's just validation library, yet it allows to do so complex functional programming with magic operators, the haskell style