Review of Python Courses (Part 5)
Posted by Mark on November 20, 2020 at 07:23 | Last modified: January 20, 2021 06:41In Part 4, I summarized my Datacamp courses 10-12. Today I will continue with the next three.
As a reminder, I introduced you to my recent work learning Python here.
My thirteenth course was Pandas Foundations. This course covers:
- Review of pandas DataFrames
- Building DataFrames from scratch
- Importing and exporting data [pd.read_csv() args: header, names, na_values, parse_dates]
- Plotting (arrays, series, DataFrames) with pandas
- Visual exploratory data analysis (line, scatter, box plots, histogram, and different plotting idioms)
- Statistical exploratory data analysis
- Separating populations
- Indexing time series (creating and using a Datetime index)
- Resampling time series data
- Manipulating time series data
- Time series visualization (pandas, not matplotlib)
- Reading and cleaning the data (cleaning and tidying datetime data)
Class #14 for me was Statistical Thinking in Python (Part 1). This course covers:
- Introduction to exploratory data analysis
- Plotting a histogram
- Plot all of your data: Bee swarm plots [sns.swarmplot()]
- Plot all of your data: Empirical Cumulative Distribution Functions (ECDF)
- Introduction to summary statistics: the sample mean and median
- Percentiles, outliers, and box plots
- Variance and standard deviation
- Covariance and the Pearson correlation coefficient
- Probabilistic logic and statistical inference
- Random number generators and hacker statistics
- Probability distributions and stories: the Binomial distribution (binomial PMF and CDF)
- Poisson processes and the Poisson distribution
- Probability density functions
- Introduction to the Normal distribution
- The Normal distribution: properties and warnings
- The Exponential distribution
My fifteenth course was Introduction to Data Visualization with Matplotlib. This course covers:
- Adding data to axes
- Customizing your plots (adding markers, setting linestyle, color, axis labels)
- Small multiples with plt.subplots
- Plotting time-series data (using fig/ax, zooming in on datetime range)
- Plotting time-series with different variables (using twin axes, coloring vars and ticks, all-encompassing function)
- Annotating time-series data
- Quantitative comparisons: bar charts (stacking, adding legend, color)
- Quantitative comparisons: histograms
- Statistical plotting
- Quantitative comparisons: scatter plots (encoding time by color)
- Preparing your figures to share with others (choosing plot style)
- Sharing your visualizations with others [fig.savefig()]
- Automating figures from data
Planning My Next Meetup [hopefully not MIS] Adventure (Part 10)
Posted by Mark on November 17, 2020 at 07:03 | Last modified: May 12, 2020 11:11I’ve been getting more organized this year by converting incomplete drafts into finished blog posts. This was the start of a nine-part mini-series. Not all of this makes sense to me now but just in case someone out there can possibly benefit, what follows is Part 10 from June 2017.
—————————
Last time, I restructured the Meetup pitch to focus on the research rather than the actual backtesting. Along these lines, I have one more idea that will conclude this mini-series of ideas (for the time being, at least).
I am a full-time, independent trader who is looking for collaboration. I am looking for someone to work with me on building an automated backtester and conducting option research. I have already done a lot of work in designing the research plan.*
Retirement savings: what are you going to do about it? Many people think “financial advisor” as a first response. When it comes to honesty in the workplace, though, some professions have a better reputation than others. Surveys over the last couple years have seen “stockbrokers” and “bankers” near the bottom (health care practitioners and grade school teachers near the top). What about financial advisors? A 2018 Harvard study found one in 12 financial advisors have been disciplined for serious misconduct with the bad apples rarely punished. You probably don’t have to ask around for long to find a salty taste in the mouth of someone who feels taken advantage of or defrauded by such deceptive promises at one time or another.
Personally, I think financial advisors can do a lot of good and deserve proper compensation. I am not convinced that they should manage your investments. For the self-starters out there, I think you can do better.
I am a Doctor of Pharmacy, but for the last 11 years I have worked for myself trading for a living.
One thing I find sorely missing from the financial industry is hardcore research and testing of investment strategies. The healthcare equivalent would be “evidence-based medicine.” I want to trade strategies that have worked in the past when I think they have reason to continue working in the future.
With regard to investing, why hire a financial advisor and pay 1-2% of your assets under management for strategies they have not personally backtested and studied? Without the data, recommendations are nothing more than hollow promises, sales, and marketing. I see a lot of this being presented.
I have spent thousands of manual hours backtesting trading strategies. With much more left to study, I seek a quicker solution: an automated backtesting app. I suggest a collaboration: you bring the programming muscle and statistics/data science background and I bring the trading know-how and experience. Let’s find out how the data suggest we invest.
While the exact percentage and time frame varies, numerous writings cite ~90% of all traders fail in their first couple years. I bring an alternative approach, though—one that may raise some eyebrows if your investment education is based on guru talk, financial media, or classic books.
Let’s work together: I help teach you about trading and you help build the app, which will benefit us both. This is your chance to get a fresh take from an industry outsider who ironically has more trading experience than the vast majority of financial professionals. Let’s do the work, make some money, and have a good time establishing Meetup community along the way!
* — I haven’t written in detail about why automated backtesting,
but the second paragraph here probably explains it.
Review of Python Courses (Part 4)
Posted by Mark on November 12, 2020 at 07:08 | Last modified: January 19, 2021 10:18In Part 3, I summarized the 6th through 9th Datacamp courses I took. Today I will continue with the next three.
As a reminder, I introduced you to my recent work learning Python here.
Course #10 was Introduction to Databases in Python. This course covers:
- Databases consist of tables
- Connecting to your database
- Introduction to SQL queries
- Filtering and targeting data
- Ordering query results
- Counting, summing, and grouping data
- SQLAlchemy and pandas for visualization
- Calculating values in a query
- SQL relationships
- Working with hierarchical tables
- Handling large ResultSets
- Creating databases and tables
- Inserting and updating data into a table
- Deleting data from a database
- Census case study
- Populating and querying the database
Course #11 was Introduction to Statistics in Python. This course covers:
- What is statistics?
- Measures of center
- Measures of spread and outliers (from scipy stats import iqr)
- What are the chances [probability and dataframe .sample() method from random module]?
- Discrete and continuous distributions
- Generating random numbers according to uniform distribution (from scipy stats import uniform)
- Computing cumulative distribution functions
- Binomial distribution (from scipy stats import binom)
- Normal distribution (from scipy stats import norm)
- Central limit theorem
- Poisson distribution (from scipy stats import poisson)
- Exponential distribution (from scipy stats import expon)
- Student’s t-distribution
- Log-normal distribution
- Correlation (and caveats)
- Experimental design and confounders
My twelfth course was Introduction to Data Visualization in Python. This course covers:
- Plotting multiple graphs
- Customizing axes
- Legends, annotations, and styles
- Working with 2-D arrays and meshgrid
- Visualizing bivariate functions (color bar, color map, axis tight, and contour plots)
- Visualizing bivariate distributions (rectangular and hexagonal binning)
- Working with images
- Visualizing regressions [sns.lmplot(), hue, col, sns.residplot()]
- Visualizing univariate distributions [sns.stripplot(), sns.swarmplot(), sns.violinplot()]
- Visualizing multivariate distributions [sns.jointplot(), kde, sns.pairplot(), hue, covariance sns.heatplot()]
- Visualizing time series (formatting datetime index)
- Time series with moving windows
- Histogram equalization in images
Networking Call (Part 1)
Posted by Mark on November 9, 2020 at 07:28 | Last modified: June 25, 2021 12:29I came across GP on a trading forum where he shared some free trader tools. I viewed his website to find:
> GP is an engineer with a deep passion for the financial markets….
Me too!
> He loves playing with numbers and got hooked onto derivative trading
> in his final years of grad school.
I got hooked a bit later but otherwise, me too!
> Over the years he has traded as a Series-7 licensed… trader…
Unlike me, he has actual experience working in the industry.
> …He also briefly served in a market data and analytics consulting
> role… advising various clients including hedge funds, commodity
> trading firms and exchanges…
This sounds great because I have thought about pursuing some sort of consulting role where I work with portfolio managers to design and/or teach trading strategies, etc.
> GP received… MS in Computer Networking and Telecommunications…
> and an inter-disciplinary Ph.D in Computer Science and
Unlike me, it sounds like he has extensive programming experience.
> Telecommunications… where he found a deep interest in probability
> theory while working on sample path analysis of stochastic processes
> using linear algebraic queueing theory as the main analytical tool.
Sounds like he has some additional financial engineering (see second-to-last paragraph here) expertise.
> He has traded extensively in Stocks, Options, Futures, Options on
> Futures and has even dabbled into Bonds and Commodties – trading
> them using various technical and fundamental models. Currently he
> mostly trades index and stock options using delta neutral strategies.
Sounds like he has extensive trading experience!
In a nutshell, then, he sounds like a full-time trader with actual industry experience: something I periodically consider. He seems to have more advanced analytical knowledge to get the job done (I have considered going back to pursue an MSFE or computer science degree). He markets himself as a consultant and has apparently worked as an industry strategy consultant.
GP sounds like a good person with whom to speak. He can probably identify with much of my experience and he may have some ideas how to get around road blocks that I have encountered (i.e. here and fourth-to-last paragraph here).
In combination with my experience, I thought we could have a good networking call. I have an advanced trading background along with at least five years spent probing the industry for potential entry points. Maybe I could help him, too.
I contacted him via e-mail through his website and we scheduled a call for last night at 5:00 PM.
I called promptly at 5 PM only to be greeted by someone who had no idea who I was. It took a good 20 seconds of silence before he said “did you find me through [the online forum]?”
“Yes.” In other words, yes that’s me… who else would it be when you agreed to schedule a call for this exact time?
Hopefully he just had a rough day.
I will continue next time.
Categories: Networking | Comments (0) | PermalinkReview of Python Courses (Part 3)
Posted by Mark on November 6, 2020 at 07:47 | Last modified: January 14, 2021 15:02In Part 2, I summarized the next three Datacamp courses I took. Today I will continue with courses 6-8.
As a reminder, I introduced you to my recent work learning Python here.
Course #6 was Python Data Science Toolbox, Part 2. This course covers:
- Iterators and iterables
- List and dictionary comprehensions
- Conditionals in comprehensions
- Generator expressions
- Using pandas read_csv iterator for streaming data
The next course I took was Merging Dataframes with pandas. Concepts covered in this course include:
- Reading multiple data files (including use of glob)
- Reindexing dataframes
- Arithmetic with series and dataframes
- Appending and concatenating series and dataframes
- Example of reading multiple files at once with a loop to build dataframe
- Concatenation, keys, and MultiIndexes
- Outer and inner joins
- Converting to, manipulating as, and stacking arrays
- Merging and joining dataframes
- Reshaping and plotting
My course #8 was Introduction to Importing Data in Python. Concepts covered here include:
- Importing text and tabular data
- Flat files
- Importing flat files using NumPy and pandas
- Importing pickled files and Excel spreadsheets
- Importing SAS/Stata files using pandas
- Importing HDF5 and MATLAB files
- Introduction to relational databases
- Creating a database engine in Python
- Querying relational databases in Python and directly with pandas
- Advanced querying: exploiting table relationships
As a bonus, let’s recap my class #9: Intermediate Importing Data with Python. Concepts covered here include:
- Importing flat files from the web with the urllib package
- HTTP requests to import files from the web
- Scraping the web in Python using BeautifulSoup
- Introduction to APIs and JSONs
- APIs and interacting with the world wide web
- The Twitter API and authentication
I will continue summarizing classes later.
Categories: Python | Comments (0) | PermalinkAutomated Backtester Research Plan (Appendix B)
Posted by Mark on November 3, 2020 at 07:44 | Last modified: May 12, 2020 14:33From Jan 2019, this should be the final entry of the current blog mini-series (see Appendix A introductory remarks).
—————————
More relevant to the WF approach, could we shorten the period to X months and look at the average and extremes of the distribution of daily price changes? We would be stopped out going into a big change. When placing trades in the new market environment placing trades, we should get compensated by higher option premiums. This gives us a fighting chance and if we can somehow decrease position size in accordance with ATR, then maybe we can keep the average PnL changes somewhat constant across anything from calm to volatile market environments. Stuff to think about.
I don’t believe this holds any relevance with regard to the daily/serial trades categorization (see Part 5 paragraphs 2-3). This may be something to study with regard to allocation or it may be something to monitor before trade entry. A study of ATR versus future MAE may help to determine what relationship (if any) exists between the two.
If the market goes up more than it goes down, then why not trade bullish butterflies rather than naked puts? Position sizing aside, the risk is much lower for the butterflies.
My initial thought on this was to study MAE and MFE of the underlying over the next ~30 days. Mild MAE numbers and better MFE would signify a market moving gradually higher. The key for butterfly profitability is when the market gets inside the expiration tent, which would not be indicated by this study. If we target something like 10%/-20% PT/max loss, though, then any movement up will likely get close to the PT unless the market skyrockets.
We could also study these max excursion distributions over different time intervals (e.g. 5-20 days by increment of 5). The butterfly would be indicated if MAE remains constant at some point while MFE continues to grow. Another indication would be a high percentage of cases showing MAE to be contained while MFE is large. Another indication would be MFE limited and positive within a certain time range. A final indication would be tip of bell curve corresponding to a set range. We could study butterflies that are centered X% above the money to X% below the money and calculate differences in profit.
We could also look at the asymmetrical butterfly (butterfly + PCS), but spread width would have to be specified, etc.
As backdrop from the easier scenario against which to compare all recent discussion (this blog mini-series), consider an all-equity portfolio where backtesting can easily be done with fixed position size throughout as defined by:
Number of shares traded = (initial number shares * initial stock price) / current stock price
Normalized drawdown can then be calculated at any point as a percentage of the initial account value.
Categories: Backtesting | Comments (0) | PermalinkAutomated Backtester Research Plan (Appendix A)
Posted by Mark on October 29, 2020 at 07:19 | Last modified: May 12, 2020 12:10I’ve been getting more organized this year by converting incomplete drafts into finished blog posts. I thought I had wrapped up the automated backtester mini-series here, but I was wrong! I have one more draft with research notes on potential future directions. On the off chance someone out there can possibly benefit, from Jan 2019 I present the next two posts.
—————————
With regard to iron condors (split-strike butterflies), maybe we set short strikes at 10-40 delta by increments of 10 and figure out what the wing widths are going to be. Maybe we close an offended vertical if short delta increases by X or if trade gets down 6% when using profit/loss targets of 10%/-15% or so. Maybe we have to calculate max potential profit and look to collect 50-90% of that as early management (or closing at 7-21 DTE by increments of 7).
From the previous post on vertical spreads we could go in a couple different directions. I talked about one specific trading plan called “The Bull.” I would like to backtest a few other particular trades (i.e. Netzero, STT, RC). I could give some particulars about those trades or maybe the backtesting plan just like I did for The Bull.
Finally, and many of these could be posted under “additional considerations” (along with many of my non-automated-backtester backtesting ideas), I have to face the possibility that all of this is done in vein. We’re looking at historical data and possibly setting critical levels that only get breached in 5% of cases. This may make me feel more comfortable when taking action at these points, but it certainly is no guarantee they will be effective in case we are in a longer-term period where distribution on that parameter changes. To that end, it would be nice if I could somehow split the data and do some WFA but I fear the sample size may be too small (see third paragraph here).
This concern even applies to something as general as daily option-price changes. Looking over the whole data span, option prices generally only change by $X/day. When IV picks up and backwardation occurs with huge ATRs, though, those daily price changes are going to be magnified—possibly 5-10X or more. Clearly this indicates a time to step out but what if such volatile activity persists for a periodi of years? This trading approach might be on the sidelines for the duration.
I will conclude next time.
Categories: Backtesting | Comments (0) | PermalinkHot Take: Insurance is Overpriced
Posted by Mark on October 26, 2020 at 07:01 | Last modified: January 9, 2021 15:48I’ve been getting more organized this year by converting incomplete drafts into finished blog posts. Given the stock market crash Q1 of this year, here is a timely piece from October 2019.
—————————
In any industry, insurance is overpriced. How do we know? Because insurance is known as a great business to be in (ask Warren Buffett). Whether or not they realize it, people have difficulty pricing the value of protecting against huge loss. As a result, they tend to overpay. Insurance gets dismissed as just another bill to pay.
With regard to cars, houses, and medical (until recently), insurance is a legal requirement. Such demand probably maintains or drives the price of insurance higher.
What if people were required to buy insurance for their stock investments? Some believe put insurance to be chronically overpriced. If people had to buy then the IV would presumably be even higher, which would be edge to the sellers. Why isn’t this the case? Only people with dispensable income (savings to invest) get involved in the stock market in the first place. This amounts to a relatively small proportion of people compared to the proportion of those with cars, homes, or medical needs.
I assume that the “smart money” such as pension funds, hedge funds, and institutional fund owners are those who predominantly buy put protection to keep the IV inflated. What a boon to business it would be if it ever came to pass that everyone had to buy!
Maybe it’s not such a bad idea to be like the “smart money” and buy the protection anyway.
Some strongly argue for this NOT to be the case. And if you have the extra money to lose, which synthetically takes the form of trading small, having lots of cash on the sidelines, and/or being relatively deleveraged, then it’s probably best not to buy the insurance at all and collect interest on those payments rather than seeing them go down the drain in the first place.
Categories: Option Trading | Comments (0) | PermalinkLead Trader and Research Assistant (Part 2)
Posted by Mark on October 23, 2020 at 07:26 | Last modified: January 9, 2021 15:31I recently wrote about a second job that seemed very appealing as a formal introduction to working in the financial industry. My cover letter is below.
——————————–
To Whom it May Concern,
I am writing to apply for the position of Lead Trader and Research Assistant.
After working as a staff pharmacist and pharmacy manager, I retired at the age of 36 to start a securities trading business. This has been a journey without clients or co-workers that has required extensive self-study, intrinsic motivation, and outside-the-box thinking. I have since learned a great deal about the mechanics of trading and investing. Having risked my own hard-earned capital to learn the craft, I now seek to manage wealth for others.
I feel my experience is something of a contradiction. I have never worked publicly for a financial firm, but I probably have more trading and portfolio management experience than the vast majority of industry participants. Trading has been my sole source of income for the last 12 years.
I have given extensive consideration to a potential career in the financial industry. I have researched the possibility of starting my own hedge fund, working as an IAR, or managing money for a family office.
I suspect the kind of trading I do for myself would not be appropriate for most clients of an IA. Accredited investors aside, I may never trade an option for the firm’s clients due to suitability standards. If options can be used in client portfolios, then I would be very eager to work together in designing viable strategies. This is not my expectation, however.
Regardless of derivative suitability, one topic for discussion is whether my nearly 20-year experience trading stock and derivatives can be marketable for the firm. I want to believe it can and would be very interested in talking this through.
My journey as a full-time trader has [by necessity] put me on a quant-related path. Over the last two years, I have invested in additional education to better understand and immerse myself in algorithmic strategy development. Overfitting is a nemesis that requires constant surveillance. A curve-fit system will look great and convince casual observers, but performance at the hard-right edge is likely to disappoint.
Aside from developing trading strategies, I hope to soon research potential benefits of combining asset classes. While “diversification” has been a prized industry buzzword, I am not sure it is as simple as being plucked out of thin air. Backtesting can be done with Excel VBA, but I think greater potential exists with Python—–a versatile programming language I have made decent headway getting to know in recent months.
As with options, my foray into algorithmic futures trading may not be suitable for client accounts. The work certainly falls in the realm of both research (backtesting) and trading (“Lead Trader and Research Assistant”). Although not expected, I would jump at the opportunity to work with a team in developing diversified futures portfolios.
I am willing to pursue further credentials that may be valuable to the firm in its dealings with clients. I have passed the Series 65 twice and am no stranger to exam preparation (Pharm.D. graduation and NAPLEX). On my own, pursuit of further credentials (e.g. CFA, MSFE, CFP) has been difficult to justify since I have been able to cover living expenses on trading profits alone. As the most highly regarded sign of success for a retail trader, this achievement makes me both thankful and proud.
I have attached a résumé for a more detailed review of my education and experience. My e-mail is {1}, and my phone is {2}. Thanks for your time and consideration!
Categories: About Me | Comments (0) | PermalinkReview of Python Courses (Part 2)
Posted by Mark on October 20, 2020 at 07:58 | Last modified: January 12, 2021 16:45In Part 1, I summarized the first two Datacamp courses I took. Today I will continue with the next three.
As a reminder, I introduced you to my recent work learning Python here.
Course #3 was Data Manipulation with pandas. This course started out with:
- Aspects of dataframes
- Sorting and subsetting
- Adding new columns
- Calculating summary statistics
- Counting (categorical data)
The course continues on with:
- Grouped Summary Statistics
- Pivot tables
- Explicit indexes
- Slicing and subsetting with .loc and .iloc accessors
- Data visualization
- Missing values
- Reading and writing .csv files
The next course I took was Data Types for Data Science in Python. Concepts covered in this class include:
- Lists (and iteration)
- Tuples (and iteration)
- Sets for unordered and unique data
- Using dictionaries
- Altering dictionaries
- Working with .csv files
- Counting with Collections module
- Dictionaries of unknown structure (defaultdict)
- Maintaining dictionary order with orderedDict
- Named tuples
- Working with datetime objects
- Adding and subtracting time
- Pendulum library
- Counting crimes case study
The next course I took was Python Data Science Toolbox, Part 1. Concepts covered in this class include:
- User-defined functions
- Multiple parameters and return values
- Object scope
- Nested functions
- Default and flexible arguments
- Lambda functions
I will continue summarizing classes later.
Categories: Python | Comments (0) | Permalink