sshoogr for your infrastructure automation for gr8conf 2017
TRANSCRIPT
01
02
Let's start!03
Background04
BackgroundBig projects built by Ant, Maven, and eventually Gradle
Teams composed mostly of Java developers
Complex (sometimes, overengineered) architectures
Many environments (DEV, TEST, QA, SIT, UAT, PREPROD, PROD)
to support
••••
05
Problems IInfrastructure is influenced by (relatively) frequent architecture
changes (components, versions, layers)
We want our environments to be the same (or at least quite similar) to
avoid any side effects during development, testing and production
We don't want to spend hours/days/weeks on configuring each and
every new server and keeping them insync
•
•
•
06
Problems IIOperations guys are not always available (e.g. busy supporting
production systems or just not skilled enough)
Development infrastructure (Jenkins, Sonar, Version Control, Load
Testing etc.) also needs maintenance
We want to reuse experience available in our team and avoid throwing
in too many various trendy technologies that will fail our expectations
•
•
•
07
First Blood08
Ant + Gradleant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
09
Simple callant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
10
Sshoogr
11
Sshoogr featuresGroovybased SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•••
12
Sshoogr0.9.25!
13
Sshoogr usecases
14
UC: scripting
15
UC: provisioning
16
UC: testing
17
UC: tunnelling
18
UC: bridging
19
UC: IOT
20
Sshoogrusage
21
Import@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.25')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
22
DefaultsdefaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
verbose = true
showCommand = true
}
01.
02.
03.
04.
05.
06.
23
ConnectionremoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm ‐rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
24
Multiline contentremoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
25
AppendableremoteFile('/etc/motd') << 'Additional message'
new File('localFile') << remoteFile('/etc/motd')
localFile << remoteFile('/etc/motd') << 'msg'
01.
02.
03.
26
File copyingremoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
27
File copyingremoteSession {
scp {
from { remoteDir '/var/bea/domain/application' }
into { localDir "$buildDir/application" }
}
}
01.
02.
03.
04.
05.
06.
28
File copyingremoteSession {
scp {
from {
remoteFile '/etc/init.d/service1'
remoteFile '/etc/init.d/service2'
}
into { localDir "$buildDir/application" }
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 29
File copyingremoteSession {
folders.each { folder ‐>
exec "zip ‐9 ‐q ‐r /tmp/${folder}.zip " +
"/usr/.../storage/${folder}"
scp {
from { remoteFile("/tmp/${folder}.zip") }
into { localDir('./repos/') }
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.30
File copyingscpOptions {
uploadToDirectory = '/tmp'
postUploadCommand =
'sudo cp ‐R %from%/* %to% && sudo rm ‐rf %from%'
}
01.
02.
03.
04.
05.
31
Command resultdef result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ‐>
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 32
Shortcutsif (ok('/usr/bin/mycmd')) {
...
}
if (fail('/usr/bin/othercmd')) {
...
}
if (commandOutput('ls /')) {
...
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 33
Tunnelstunnel('1.2.3.4', 8080) { int localPort ‐>
def url = "http://localhost:${localPort}/flushCache"
def result = new URL(url).text
if (result == 'OK') {
println "Cache is flushed!"
} else {
throw new RuntimeException(result)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 34
Prefix/suffixprefix('sudo ') {
exec 'rm ‐rf /var/log/abc.log'
exec 'service abc restart'
}
suffix(' >> output.log') {
exec 'yum ‐y install nginx'
exec 'yum ‐y install mc'
exec 'yum ‐y install links'
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 35
More features36
ANSI colors
37
Rainbow DemoremoteSession(
'vagrant:[email protected]:22'
) {
exec(
"bash ‐c 'yes \"\$(seq 231 ‐1 16)\" | " +
"while read i; " +
" do printf \"\\x1b[48;5;\${i}m\\n\"; " +
"sleep .02; " +
"done'")
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.38
Download progress
39
HTTP proxySSH over HTTP? Really? Yes!
defaultProxyHost=192.168.1.2
defaultProxyPort=8080
01.
02.
40
Sshoogr executablesshoogr ‐h 192.168.43.122
‐u ubuntu
‐l color
default.sshoogr
01.
02.
03.
04.
41
*.sshoogr scriptsNo need to install Groovy
Can ommit @Grab
Can ommit connection details
Scripts are more portable
Ansible for Groovy!
•••••
42
*.sshoogr scriptsremoteSession {
exec('uname ‐a')
exec('date')
...
}
01.
02.
03.
04.
05.
43
sdkman.io
44
sdkman.iocurl ‐s http://get.sdkman.io | bash
sdk install sshoogr 0.9.25
sdk list sshoogr
01.
02.
03.
45
Demo46
More goodies47
Intellij IDEADSL support
48
Intellij IDEA DSL support
49
Intellij IDEA DSL support
50
Demo51
Intellij IDEA DSL supporthttps://confluence.jetbrains.com/display/GRVY/Scripting+IDE+for+DSL+awareness•
52
SSHD Mock53
SSHD MockMockSshServer.with {
command('^ls.*$') { inp, out, err, callback, env ‐>
out << '''total 20
drwxr‐xr‐x 3 1100 1100 4096 Aug 7 16:52 .
drwxr‐xr‐x 8 1100 1100 4096 Aug 1 17:53 ..
drwxr‐xr‐x 3 1100 1100 4096 Aug 7 16:49 examples
callback.onExit(0)
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09. 54
SSHD Mock ...
command('^whoami.*$') { inp, out, err, callback, env ‐>
out << "root\n"
callback.onExit(0)
}
command('^du.*$') { inp, out, err, callback, env ‐>
out << "100\n"
callback.onExit(0)
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.55
SSHD Mock ...
// Create file expectations.
dir('.')
dir('/tmp')
...
01.
02.
03.
04.
05.
56
SSHD Mock ...
// Start server
startSshd(2233)
}
01.
02.
03.
04.
57
Demo58
PUnit59
PUnitSimple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
••••
60
PUnit: example (derby)class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
61
PUnit: example (derby)@Test
void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
62
PUnit: example (derby)@Test
void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
63
PUnit: example (derby) ...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09. 64
PUnit: example (derby) ...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
65
PUnit: example (svn)session {
tunnel ('127.0.0.1', 80) { int localPort ‐>
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1',
localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.
04.
05.
06.
07.
08.
09. 66
PUnit: example (svn) ...
// Setup credentials.
def authManager = SVNWCUtil.
createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.
06.
07.
08.
09. 67
PUnit: example (svn) ...
// Commit first revision.
ISVNEditor editor = repository.
getCommitEditor("Initial commit.", null)
editor.with {
openRoot(‐1)
addFile('dummy.txt', null, ‐1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
01.
02.
03.
04.
05.
06.
07.
08.
09. 68
PUnit: example (svn) ...
def checksum = deltaGenerator.sendDelta('dummy.txt',
new ByteArrayInputStream("data".getBytes()),
editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09. 69
PUnit: example (svn) ...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.
05.
70
PUnit: continuous integration
71
PUnit: Jenkins build
72
TAP: Test Anything Protocol1..4
ok 1 ‐ Input file opened
not ok 2 ‐ First line of the input valid
ok 3 ‐ Read the rest of the file
not ok 4 ‐ Summarized correctly
01.
02.
03.
04.
05.
73
TAP with Sshoogrdef tests = [
'sdkman should create candidates directory':
'test ‐d ~/.sdkman/candidates',
'7 groovies should be installed':
'test 7 ‐eq `ls ‐1 ~/.sdkman/candidates/groovy ' +
'| grep ‐v current | wc ‐l`'
]
01.
02.
03.
04.
05.
06.
07.
74
TAP with SshoogrremoteSession {
connect()
println "1..${tests.size()}"
tests.eachWithIndex { name, command, index ‐>
if (ok(command)) {
println "ok ${index + 1} ‐ ${name}"
} else {
println "not ok ${index + 1} ‐ ${name}"
}}}
01.
02.
03.
04.
05.
06.
07.
08.
09. 75
TAP with Sshoogr>>> Connecting to 192.168.33.144
1..2
ok 1 ‐ sdkman should create candidates directory
not ok 2 ‐ 7 groovies should be installed
<<< Disconnected from 192.168.33.144
01.
02.
03.
04.
05.
76
Demo77
Jenkins TAP plugin
78
Gradleintegration79
Gradle integrationbuildscript {
repositories { mavenCentral() }
dependencies {
classpath 'com.a....sshoogr:sshoogr‐gradle:0.9.18'
}
}
apply plugin: 'secureShell'
01.
02.
03.
04.
05.
06.
07.
80
Gradle integrationtask remoteTask << {
remoteSession("user:password@localhost:22") {
exec 'rm ‐rf /tmp/cache/'
scp "$buildDir/cache.content",
'/tmp/cache/cache.content'
}
}
01.
02.
03.
04.
05.
06.
07.
81
Gradle integrationsshOptions {
defaultUser = remoteShellUser
defaultPassword = remoteShellPassword
}
01.
02.
03.
04.
82
gradle.properties (project's home)remoteShellUser=<PLEASE SET ME>
remoteShellPassword=<PLEASE SET ME>
01.
02.
83
gradle.properties (user's home)remoteShellUser=andrey
remoteShellPassword=yes_it_is_my_real_password
01.
02.
84
Demo85
Little brother86
Groowin@Grab('com.aestasit.infrastructure.groowin:groowin:0.1.8')
import static com.aestasit.winrm.DefaultWinRM.*
01.
02.
87
GroowinremoteManagement {
host = '127.0.0.1'
user = 'Administrator'
password = 'secret'
exec 'del', 'C:\\temp.txt'
remoteFile('C:\\my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
07.
88
Summary89
Sshoogr is...Battletested Groovy DSL for SSH connectivity
Executable and portable scripting tool
Easily integratable with any Java/Groovy library
•••
90
Sshoogr can be used for...Provisioning your servers and IoT devices
Executing remote orchestration commands
Testing and monitoring infrastructure state
•••
91
Next stepsResource definitions
Command rollbacks
Parallel execution
XSS utilities
Extend integration tests
Better documentation
••••••
92
Seekingcontributors!
93
Source codeSshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogrgradle
Groowin: https://github.com/aestasit/groowin
Groowin Gradle: https://github.com/aestasit/groowingradle
••••
94
Source codeSshdmock: https://github.com/aestasit/groovysshdmock
Punit: https://github.com/aestasit/punit
WinRM client: https://github.com/aestasit/groovywinrmclient
•••
95
Thank you!96
97