HG_REPOSITORIES/LPP/INSTRUMENTATION/USERS/JEANDET/presentations/LoOPs_TeamCity Files · LoOPS_TeamCity.ipynb

Final version plus some cleaning...

Final version plus some cleaning Signed-off-by: Alexis Jeandet <alexis.jeandet@member.fsf.org>

jeandet - - Load All Authors

File last commit:

r2:fbb8b7d9e310 master


                r2:fbb8b7d9e310

master

Download file

             LoOPS_TeamCity.ipynb
        
                    517 lines
            
             | 14.3 KiB
            
                | text/plain
            
             |
                TextLexer

/ LoOPS_TeamCity.ipynb

History | Source | Raw |Copy content |Copy permalink

jeandet init...	r0	{
		"cells": [
		{
		"cell_type": "code",
jeandet Final version plus some cleaning...	r2	"execution_count": 1,
jeandet init...	r0	"metadata": {
		"ExecuteTime": {
jeandet Final version plus some cleaning...	r2	"end_time": "2018-01-10T22:15:31.021568Z",
		"start_time": "2018-01-10T22:15:31.010972Z"
jeandet init...	r0	},
		"init_cell": true
		},
		"outputs": [],
		"source": [
		"from traitlets.config.manager import BaseJSONConfigManager\n",
jeandet Final version plus some cleaning...	r2	"from os.path import expanduser\n",
		"home = expanduser(\"~\")\n",
		"path = home+\"/.jupyter/nbconfig\"\n",
jeandet init...	r0	"cm = BaseJSONConfigManager(config_dir=path)\n",
		"cm.update(\"livereveal\", {\n",
		" \"theme\": \"solarized\",\n",
		" \"transition\": \"zoom\",\n",
		" \"start_slideshow_at\": \"selected\"\n",
		"});\n"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# TeamCity\n",
		"\n",
jeandet Final version plus some cleaning...	r2	"<div>\n",
jeandet init...	r0	"<center>\n",
jeandet Final version plus some cleaning...	r2	"<img src=\"resources/teamcity-logo.png\" width=\"300px\" align=\"center\">\n",
		"</center>\n",
		"</div>\n",
		"\n",
		"\n",
		"\n",
		"<p align=\"center\">\n",
		" <b>Alexis Jeandet <alexis.jeandet@lpp.polytechnique.fr>:</b><br>\n",
		" <a href=\"https://github.com/jeandet\">Github:jeandet</a> \|\n",
		" <a href=\"https://gitter.im/jeandet\">Gitter:jeandet</a> \|\n",
		" <a href=\"https://freenode.net/\">IRC@freenode:jeandet</a>\n",
		" <br>\n",
		"</p>\n",
		"\n",
		"\n",
		"\n"
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
jeandet Final version plus some cleaning...	r2	"# TC quick overview\n",
jeandet init...	r0	"\n",
		"* A powerful distributed Continuous Integration tool\n",
		"* Rich plugin ecosystem\n",
		"* Really customizable system\n",
		"* Not so complex...\n",
jeandet Ready for talk...	r1	"* Updates so easy (Compared to redmine...)\n",
jeandet init...	r0	"* Mostly written in Java\n",
jeandet Ready for talk...	r1	"* Closed source :(\n",
		"* Limited to 100 build cfg & 3 agents "
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# TC Plugins (335 on [JetBrains website](https://plugins.jetbrains.com/teamcity) )\n",
		"\n",
		"* Web Hooks plugin \n",
		"* AWS S3 Artifact Storage \n",
		"* Google Cloud Agents \n",
		"* Docker Cloud \n",
		"* Debian Package Server \n",
		"* Slack Build Notifier\n",
		"\n",
		"\n",
		" Installation as simple as uploading a zip file! \n",
		"\n"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"ExecuteTime": {
		"end_time": "2018-01-02T20:14:04.757140Z",
		"start_time": "2018-01-02T20:14:04.749828Z"
		},
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# Vocabulary\n",
		"\n",
		"* Agent: <span style=\"color:blue\">where stuff is done</span>"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Build step: <span style=\"color:blue\">the smallest build task subdivision</span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editBuildRunners.html?id=buildType:SciQLop_UnitTests) ) "
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Build config: <span style=\"color:blue\">a collection of build steps </span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editBuild.html?id=buildType:SciQLop_UnitTests) )"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
jeandet Final version plus some cleaning...	r2	"* VCS root: <span style=\"color:blue\">a mostly a VCS URL + some options </span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editVcsRoot.html?action=editVcsRoot&vcsRootId=SciQLop_HttpsHephaistosLppPolytechniqueFrRhodecodeHgRepositoriesLppSciQLOPReposS&editingScope=buildType%3ASciQLop_UnitTests&cameFromUrl=%2Fteamcity%2Fadmin%2FeditBuildTypeVcsRoots.html%3Finit%3D1%26id%3DbuildType%253ASciQLop_UnitTests&cameFromTitle=Edit%20Build%20Configuration) )"
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Project: <span style=\"color:blue\">a collection of build configs + VCS + settings </span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editProject.html?projectId=SciQLop) )"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
jeandet Ready for talk...	r1	"slide_type": "subslide"
jeandet init...	r0	}
		},
		"source": [
jeandet Ready for talk...	r1	"# Vocabulary\n",
		"\n",
jeandet init...	r0	"* Build artifact: <span style=\"color:blue\">a file or a folder produced during the build </span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/overview.html) )"
		]
		},
jeandet Ready for talk...	r1	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"\n",
		"\n",
		"\n",
		"\n",
		"\n",
		"* Meta-Runners: <span style=\"color:blue\">a build config that has been extracted to make a template </span> ( [example](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editProject.html?projectId=_Root&tab=metaRunner) )"
		]
		},
jeandet init...	r0	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# LPP goals (What we want)\n",
		"\n",
		"\n",
		"* Being able to build on different OS"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"ExecuteTime": {
		"end_time": "2018-01-08T19:36:05.070215Z",
		"start_time": "2018-01-08T19:36:05.066706Z"
		},
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Build inside a clean environment"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"ExecuteTime": {
		"end_time": "2018-01-08T19:36:05.399625Z",
		"start_time": "2018-01-08T19:36:05.394453Z"
		},
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Produce compatibility matrices"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Being faster than Travis"
		]
		},
jeandet Ready for talk...	r1	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "subslide"
		}
		},
		"source": [
		"# LPP goals (What we want)"
		]
		},
jeandet init...	r0	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Customize workflow"
		]
		},
jeandet Ready for talk...	r1	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "fragment"
		}
		},
		"source": [
		"* Give users maximum freedom"
		]
		},
jeandet init...	r0	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# LPP Setup\n",
		"<center>\n",
		"<img src=\"resources/Setup LPP.png\" width=\"60%\" align=\"center\">\n",
		"</center>"
		]
		},
		{
		"cell_type": "markdown",
jeandet Ready for talk...	r1	"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
jeandet init...	r0	"source": [
		"# Available build agents\n",
		"\n",
jeandet Ready for talk...	r1	"* Based on [Docker cloud plugin](https://plugins.jetbrains.com/plugin/9306-docker-cloud)\n",
jeandet init...	r0	"* Virtually anything that runs either inside Docker or VirtualBox(Vagrant)\n",
		"* Spawn on demand\n",
		"* For now Windows, Linux(Fedora, Ubuntu), FreeBSD and Mac OSX 10.12.2(Siera)\n"
		]
		},
jeandet Ready for talk...	r1	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# FreeBSD & Mac \n",
		"\n",
		"[meson@lpp](https://hephaistos.lpp.polytechnique.fr/teamcity/viewLog.html?buildId=6388&tab=buildLog&buildTypeId=mesonbuild_Build_2&logTab=tree&filter=all) [meson@travis](https://travis-ci.org/mesonbuild/meson/builds/325859215?utm_source=github_status&utm_medium=notification)\n",
		"\n",
		"[OSX agent](https://github.com/jeandet/teamcity-docker-agent-osx) [FreeBSD agent](https://github.com/jeandet/teamcity-docker-agent-freebsd)\n",
		"\n",
		"* Vagrant inside Docker\n",
		"* Few txt files produces several GB machines\n",
		"* Easy to deploy\n",
		"* Hard to debug\n"
		]
		},
jeandet init...	r0	{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# I want to choose where I get built!\n",
		"\n",
		"Team city introduce the notion of build requirements.\n",
		"\n",
jeandet Ready for talk...	r1	"- Agents expose [parameters](https://hephaistos.lpp.polytechnique.fr/teamcity/agentDetails.html?agentTypeId=72&tab=agentParameters)(<=> properties). ( [sources](https://github.com/jeandet/teamcity-docker-complete-agent/blob/master/Dockerfile#L36) )\n",
jeandet init...	r0	"\n",
jeandet Ready for talk...	r1	"- For each build config, ones can declare some [requirements](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editRequirements.html?id=buildType:Miniphare_Build) that the agent has to match to get spawned."
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
jeandet Ready for talk...	r1	"# What about reports?\n",
		"\n",
		"* As build artifact ( [LFR](https://hephaistos.lpp.polytechnique.fr/teamcity/viewLog.html?buildId=292&tab=artifacts&buildTypeId=LfrFlightSoftware_Build), [Hephaistos Web](https://hephaistos.lpp.polytechnique.fr/teamcity/project.html?projectId=HephaistosWeb&tab=projectOverview), [LibCDF](https://hephaistos.lpp.polytechnique.fr/teamcity/project.html?projectId=LibCDF&tab=projectOverview) )\n",
		"* As Mail\n",
		"* As slack message\n",
		"* As stdout values ( [GStreamer Ninja](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editRunType.html?id=buildType:mesonbuild_GStreamerNightly&runnerId=RUNNER_69&cameFromUrl=%2Fteamcity%2Fadmin%2FeditBuildRunners.html%3Fid%3DbuildType%253Amesonbuild_GStreamerNightly%26init%3D1&cameFromTitle=) )\n",
		"* As XML File ( [LibCDF GTest](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editBuildFeatures.html?id=buildType:LibCDF_Build) )"
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
jeandet Ready for talk...	r1	"# Interactions with other tools\n",
		"\n",
		"* SonarQube ([example](https://sonarcloud.io/dashboard?id=sciqlop), [config](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editRunType.html?id=buildType:SciQLop_Build&runnerId=RUNNER_22&cameFromUrl=%2Fteamcity%2Fadmin%2FeditBuildRunners.html%3Fid%3DbuildType%253ASciQLop_Build%26init%3D1&cameFromTitle=))\n",
		"* Slack ([example](https://sciqlop.slack.com/messages/C57HU1KBJ/), [config](https://hephaistos.lpp.polytechnique.fr/teamcity/slacknotifications/index.html?projectId=SciQLop))\n",
		"* Github, Bitbucket, VS Team Services...\n",
		"* Rhodecode ([Miniphare PR](https://hephaistos.lpp.polytechnique.fr/rhodecode/GIT_REPOSITORIES/LPP/phare/miniphare/miniphare/pull-request/410), [config](https://hephaistos.lpp.polytechnique.fr/teamcity/admin/editRunType.html?id=buildType:Miniphare_Build&runnerId=RUNNER_82&cameFromUrl=%2Fteamcity%2Fadmin%2FeditBuildRunners.html%3Fid%3DbuildType%253AMiniphare_Build%26init%3D1&cameFromTitle=))\n",
		"* [VS IDE plugin](https://confluence.jetbrains.com/display/TCD10/Visual+Studio+Addin), [Eclipse plugin](https://confluence.jetbrains.com/display/TCD10/Eclipse+Plugin), [IntelliJ-based IDEs](https://blog.jetbrains.com/teamcity/2017/10/teamcity-integration-with-intellij-based-ides/)"
jeandet init...	r0	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
jeandet Ready for talk...	r1	"# API?\n",
		"\n",
		"* Quite easy to [understand](https://confluence.jetbrains.com/display/TCD10/REST+API) ( [buildTypes](https://hephaistos.lpp.polytechnique.fr/teamcity/app/rest/buildTypes/), [projects](https://hephaistos.lpp.polytechnique.fr/teamcity/app/rest/projects/) )\n",
		"* Almost everything can be done through API"
jeandet init...	r0	]
jeandet Ready for talk...	r1	},
		{
		"cell_type": "code",
jeandet Final version plus some cleaning...	r2	"execution_count": 38,
jeandet Ready for talk...	r1	"metadata": {
		"ExecuteTime": {
jeandet Final version plus some cleaning...	r2	"end_time": "2018-01-10T22:12:48.694242Z",
		"start_time": "2018-01-10T22:12:48.466252Z"
jeandet Ready for talk...	r1	},
		"slideshow": {
		"slide_type": "slide"
		}
		},
jeandet Final version plus some cleaning...	r2	"outputs": [
		{
		"name": "stdout",
		"output_type": "stream",
		"text": [
		"{\n",
		" \"id\": \"_Root\",\n",
		" \"name\": \"<Root project>\",\n",
		" \"description\": \"Contains all other projects\",\n",
		" \"href\": \"/app/rest/projects/id:_Root\",\n",
		" \"webUrl\": \"https://hephaistos.lpp.polytechnique.fr/teamcity/project.html?projectId=_Root\"\n",
		"}\n"
		]
		}
		],
jeandet Ready for talk...	r1	"source": [
jeandet Final version plus some cleaning...	r2	"import teamcity\n",
		"import json\n",
		"\n",
		"print(json.dumps(teamcity.tc.projects()[0].__dict__, indent=1))"
jeandet Ready for talk...	r1	]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# What Next?\n",
		"\n",
		"* Improve Rhodecode integration\n",
		"* Boost agents startup time\n",
		"* Switch to Docker swarm\n",
		"* More Meta-Runners"
		]
		},
		{
		"cell_type": "markdown",
		"metadata": {
		"slideshow": {
		"slide_type": "slide"
		}
		},
		"source": [
		"# Questions?\n"
		]
jeandet init...	r0	}
		],
		"metadata": {
		"celltoolbar": "Slideshow",
		"kernelspec": {
		"display_name": "Python 3",
		"language": "python",
		"name": "python3"
		},
		"language_info": {
		"codemirror_mode": {
		"name": "ipython",
		"version": 3
		},
		"file_extension": ".py",
		"mimetype": "text/x-python",
		"name": "python",
		"nbconvert_exporter": "python",
		"pygments_lexer": "ipython3",
		"version": "3.6.3"
		},
		"toc": {
		"nav_menu": {},
		"number_sections": true,
		"sideBar": true,
		"skip_h1_title": false,
		"title_cell": "Table of Contents",
		"title_sidebar": "Contents",
		"toc_cell": false,
		"toc_position": {},
		"toc_section_display": true,
		"toc_window_display": false
		},
		"varInspector": {
		"cols": {
		"lenName": 16,
		"lenType": 16,
		"lenVar": 40
		},
		"kernels_config": {
		"python": {
		"delete_cmd_postfix": "",
		"delete_cmd_prefix": "del ",
		"library": "var_list.py",
		"varRefreshCmd": "print(var_dic_list())"
		},
		"r": {
		"delete_cmd_postfix": ") ",
		"delete_cmd_prefix": "rm(",
		"library": "var_list.r",
		"varRefreshCmd": "cat(var_dic_list()) "
		}
		},
		"types_to_exclude": [
		"module",
		"function",
		"builtin_function_or_method",
		"instance",
		"_Feature"
		],
		"window_display": false
		}
		},
		"nbformat": 4,
		"nbformat_minor": 2
		}

	Site-wide shortcuts
/	Use quick search box
g h	Goto home page
g g	Goto my private gists page
g G	Goto my public gists page
g 0-9	Goto bookmarked items from 0-9
n r	New repository page
n g	New gist page

	Repositories
g s	Goto summary page
g c	Goto changelog page
g f	Goto files page
g F	Goto files page with file search activated
g p	Goto pull requests page
g o	Goto repository settings
g O	Goto repository access permissions settings
t s	Toggle sidebar on some pages