-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathmotd.html
More file actions
executable file
·301 lines (273 loc) · 14.5 KB
/
Copy pathmotd.html
File metadata and controls
executable file
·301 lines (273 loc) · 14.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
<!DOCTYPE html>
<html>
<head>
<title>Consumption Analytics </title>
<style>
a { color: yellow;}
hr {border-style: dashed;}
table {border: 0px; margin-left: auto; margin-right: auto;}
td {border: 3px dotted white; padding: 10px; text-align: left; vertical-align: top;}
td.empty {border: 0px; padding: 10px; text-align: left; vertical-align: top;}
#default {color: #ffffff; font-family:verdana;}
#pacman {color: #ffff00; font-family:verdana;}
#blinky {color: #ff5555; font-family:verdana;}
#pinky {color: #ff99cc; font-family:verdana;}
#inky {color: #22FAFD; font-family:verdana;}
#clyde {color: #ffcc99; font-family:verdana;}
</style>
</head>
<body style="background-color:black;">
<p><img src="https://teradata.sharepoint.com/:i:/r/teams/SalesTech/GLOBAL/Images/Banners/coa_banner_pacman2.jpg"
alt="COA Banner - PacMan"
width="100%"/></p>
<span style="color: #FFFFFF;">
<h1 id="pacman">Consumption Analytics</h1>
<p id="default">Welcome to Consumption Analytics! This newly funded project is undergoing rapid development -
please check back frequently for updates.
If you have questions, feel free to email <a href="mailto:Stephen.Hilton@Teradata.com">Stephen Hilton</a>
</p>
<table width="90%">
<tbody>
<tr>
<td style="width: 50%;" id="pinky" rowspan="2">
<!-- version updates -->
<h3><u>Version 0.4.0.7 (Sept 17, 2020)</u></h3>
<ul>
<li>updated installer, <a href="https://github.com/tdcoa/tdcsm/releases/download/v0.1.2/tdcsm-install.py">found here</a>, to handle OneDrive absconding with the Desktop folder. This installer creates a Python Virtual Environment, reducing the number of environmental conflicts and making it work more smoothly on many more machines
<li>added better end-of-process markers, for clarity (aka ########## 'reload_config' complete! ##########)
<li>fixed several bugs in the 'assisted run' use-case (aka 'Make Customer Files' and 'Process Return Data')
</ul></br>
<h3><u>Version 0.4.0.4 (Sept 10, 2020)</u></h3>
<ul>
<li>Added an installer, <a href="https://github.com/tdcoa/tdcsm/releases/download/v0.1.2/tdcsm-install.py">found here</a>, that installs application into a Python Virtual Environment, reducing the number of environmental conflicts and making it work more smoothly on many more machines
<li>Moved the requirement back to Python 3.8.x (from 3.8.5, although 3.8.5+ is still recommended)
<li>Added a real commandline utility, indepedent of Python. To get started, open up a terminal /command window and run the help command:</br><code>tdcsm -h</code>
<li>Many GUI changes to help make the experience more intuative
<li>Added an alternate path to Execute, informally called "" for those sites that don't have permission to execute directly: "Assisted Run"
<ul>
<li>Download Files and Prepare SQL, as per normal
<li>Instead of "Execute Run", users can now "Make Customer Files"
<li>This generates both SQL and BTEQ files, including a singular *.combined.sql and *.combined.bteq files, that can be handed to customers to execute
<li>Once you get .csv or .psv (pipe-delimited) files back from the customer, you can "Process Return Data" to continue processing visualizations and PPTx generation
<li>This "Process Return Data" also builds an upload manifest, so returned data can be uploaded to Transcend per the normal final step
</ul>
<li>Simplified the Teradata connection object to dramatically speed up execution of SQL (on local computer)
<li>PPTx module improved & bug fixes
</ul></br>
<!-- version updates -->
<h3><u>Version 0.3.9.6.9 (July 7, 2020)</u></h3>
<ul>
<li>New/missing Filesets now added to source_system.yaml automatically (active = False)
<li>If a required file is missing from the AppRoot (working directory), the process now looks in '0_override' folder first and will copy that version if present. Only if not found in 0_override will it pull a new copy from the python install package.
<li>Added new text-base UI: "run_cmdline." This was originally used for development/testing, but is useful enough to include as an optioinal UI.
<ul>
<li>To use: <code>   python run_cmdline.py</code>
<li>Optionally specify approot and secrets.yaml file:
</br><code>python run_cmdline.py --approot 'c:\myCust' --secrets '../!secrets.yaml'</code>
</ul>
<li>Bug fix: "KeyError: 'skip_dbs'" caused by new debugging functionality
<li>PPTx module now looks for special/replacement commands in block shapes, tables, and text blocks. If more object types are needed, please let us know.
</ul></br>
<h3><u>Version 0.3.9.6.6 (June 30, 2020)</u></h3>
<ul>
<li>Added "skip_dbs" option back in, to help tweak GUI issues more quickly
<li>Various GUI usability enhancements
<li>Bug fix to avoid failure of upload step when any of the SQL returns Zero rows
</br>This fix will skip adding entry to the upload manifest file if SQL returns zero rows
<li>Handled missing files error in the pptx module for special commands inside picture, text and table placeholders
</br>Added ** at the end of the text (or special command) to indicate that the substituted value was not replaced because the file doesn’t exist
</ul></br>
<h3><u>Version 0.3.9.6.2 (June 23, 2020)</u></h3>
<ul>
<li>COA GUI 1.0 is live! To start, simply open a terminal window / cmd window, and type or copy/paste:
<br><code>     python -c "from tdcsm.tdgui import coa;c = coa()"</code><br/>
If you haven't run COA in awhile, its recommended you run the above in an empty folder, to allow the new config file defaults to be created.
There is a known issue that the "open" buttons don't always work on Windows... that and other enhancements coming soon!
<li>Systems now hold two new elements, dbsversion and collection. For example, a system running 16.20 and pdcr would look like:
<ul>
<li>dbsversion: "16.20"
<li>collection: "pdcr"
</ul>
With these additions, we get down to one fileset (i.e., DBQL_Core) instead of one fileset per combination (i.e., DBQL_Core_1620pdcr)
<li>Download_files() only happens once, not one-per-system (i.e., more than it needed to)
<li>Fixed several bugs pertaining to file-handling differences between Mac and Windows
<li>Removed the "skip_git" and "skip_dbs" settings
<li>Fixed DBQL_Core spool-out issues
<li>Fixed dim_app.csv file encoding problem
<li>New Fileset: vantage_health_check (from APAC)
</ul></br>
<h3><u>Version 0.3.9.5.0 (June 10, 2020)</u></h3>
<ul>
<li>Connection and Encryption settings are now set at a per-system basis within source_systems.yaml
<li>ODBC connection type fixed
<ol>
<li>Driver name is case sensitive and must match EXACTLY</li>
<li>Driver names can be found in the windows ODBC Data Source Administrator under the Drivers tab</li>
</ol>
<li>Application can run special commands relating to sql that does not return any values
<li>Fixes error caused by re-processing of pptx files
<li>Fixes out-of-bounds issue relating to pptx processing
</ul></br>
<h3><u>Version 0.3.9.4.4 (June 2, 2020)</u></h3>
<ul>
<li>added fileset: db_objects
<li>defaults can now be assigned per-file (or per fileset, like before)
<li>if config.yaml // settings // write_to_perm = "True", the upload_to_transcend() process will take an
alternate upload approach to help debugging:
<ol>
<li>write table to a perm table in adlste_coa_stg
<li>insert-select into global temp table in adlste_coa_stg
<li>call the stored proc to perform merge
<li>if successful, it will drop the perm table
<li>if NOT successful, the perm table in STG persists, to allow investigation
</ul>
</td>
<td id="inky" >
<!-- version updates -->
<h3><u>Useful Links</u></h2>
<ul>
<li><a id="inky" target="_blank" href="https://teradata.sharepoint.com/:w:/r/teams/Sales/ConsumptionAnalytics/Libraries/Documentation/COA%20Python%20Project%20User%27s%20Guide%20v1.0.docx?d=w4649565c1de344328a424007858a4db7&csf=1&web=1&e=1eT4cI">Getting Starting Guide</a>
<li><a id="inky" target="_blank" href="https://app.powerbi.com/groups/6d187dfd-cff0-4365-8437-b6b49e6e76cc/reports/ec5fca03-42a1-4b14-973d-9766f669c920?ctid=9151cbaa-fc6b-4f48-89bb-8c4a71982138">PowerBI Reports (requires DBQL_Core fileset upload)</a>
<li><a id="inky" target="_blank" href="https://teradata.sharepoint.com/teams/Sales/ConsumptionAnalytics">Consumption Analytics SharePoint</a>
<li><a id="inky" target="_blank" href="https://trello.com/b/9IoDVkKH/customer-success-platform">COA Trello Project Board</a>
<li><a id="inky" target="_blank" href="https://github.com/tdcoa/sql/tree/master/filesets">COA SQL on GitHub</a>
</ul>
</br>
<h3><u>Quick INSTALL Instructions</u></h2>
<ol>
<li>Confirm Python 3.8:<code>   python --version</code>
<li>Download the installer <a href="https://github.com/tdcoa/tdcsm/releases/download/v0.1.1/tdcsm-install.py">from here</a>
<li>Run installer via commandline:<code>  python tdcsm-install.py</code>
<li>The installer will
<ul>
<li>setup a virutal environment
<li>install TDCSM
<li>create a desktop shortcut, allowing you to start COA with a double-click
</ul>
<li>To update, simply repeat these steps
</ol>
</br>
<h3><u>Quick START Instructions</u></h2>
<ul>
<li>On commandline (local environment): <code>  tdcsm</code> <br>    OR
<li>Double-click shortcut created by installer (for virtual environment -- recommended)
</ul>
</br>
<h3><u>Quick Configure Instructions</u></h2>
For running against Transcend, as a first test
<ol>
<li>Open "secrets.yaml" file (found in the app's root folder)
<li>Fill in Teradata Quicklook ID and Password, save and close
<li>'Reload Config' if you have the GUI open
<li>Make sure you're on the Teradata VPN
<li>Systems 'Transcend' and 'TranscendAWS' should both work for all FileSets - give it a test!
</ol>
</br>
<h3><u>Quick Configure Instructions</u></h2>
For running against Customer System
<ol>
<li>Open "source_systems.yaml" file, copy the 'Transcend' system definition
<li>Paste once per customer system, and change section name
<li>Validate / Change the settings:
<ul>
<li>siteid - matchs CS&S records
<li>dbsversion - 16.20, 16.00, 15.10, etc.
<li>collection - pdcr or dbc
<li>host - IP or DNS for login
<li>logmech - TD2 (default), ldap, KRB5, NTLM, etc.
<li>encryption - true / false (default)
<li>username and password - ensure the substitution values match entries found in the "secrets.yaml" file
</ul>
<li>(OPTIONAL) open "config.yaml" and update 'startdate' and 'enddate' to desired date range
<li>'Reload Config' if you have the GUI open
<li>Make sure you're on the Customer VPN
</ol>
</td>
</tr>
<tr>
<td id="clyde">
<!-- filesets -->
<h3><u>FileSet List</u></h2>
<table>
<tr>
<td class="empty">demo</td>
<td class="empty">
Simple demonstration project that exercises the major functionality.
Should work on any Teradata system, regardless of version or platform.
You will need access to Transcend to perform final upload_to_transcend()
</td>
</tr>
<tr>
<td class="empty">dbql_core</td>
<td class="empty">
Comprehensive data pull from QueryLog (dbql) table, aggregating
by application, statement type, user department, and time. Collects
CPU, IO, Query Counts, Spool, Cache Rates... all the major metrics
available in DBQL that can aggregated by metrics above.
Defaults to 6 weeks (42 days) per run.
<br>Of special note:
<ul><li>This is primary source of PowerBI project
<li>Even aggregated, this can be a large extract depending on workload diversity.
Expect in the neighborhood of 500k to 1M rows per quarter (3mo) on busy systems
</td>
</tr>
<tr>
<td class="empty">concurrency</td>
<td class="empty">
Generates concurrency average, peak, 80th and 95th percentile numbers,
by day by hour. Default is 6 weeks. This includes auto-generated visualizations,
including a line chart and heat-map.
</td>
</tr>
<tr>
<td class="empty">top_users</td>
<td class="empty">
Generates top users, per user, per week, month, and time period total.
"Top" is measured and ranked several different ways:
<ul><li>CPU
<li>IOGB
<li>Query Count
<li>Query Complexity
<li>Query RunTime
<li>Error Count (excluded in overall rank)
</ul>
These ranks are added up and re-ranked, to arrive at an overall Total Rank (except for Error Count).
When WeekID and MonthID are both null, ranks are for the time-period total. When WeekID is null, ranks are for
the month. When WeekID is NOT null, ranks are for the week. Partial weeks are not allowed, while months must
have at least 3 full weeks of data to be included.
The local "top_users.csv" file will contain UserName as well as a unique "UserHash." When uploading
to Transcend, only the non-identifiable and non-reversable UserHash will be saved, thus preventing
any customer PII from being stored on Trancend. K-Mean clustering visualizations included.
</td>
</tr>
<tr>
<td class="empty">db_objects</td>
<td class="empty">
Collection of analysis around database object characteristics, such as count of columns by types or formats,
and in near-future, other table, database, or user analysis.
</td>
</tr>
<tr>
<td class="empty">success_plan</td>
<td class="empty">
Generates a template Consumption Analytics section for the CSM Success Plan, by stringing together
other filesets, namely:
<ul><li>cpu_summary
<li>concurrency
<li>top_users
<li>...more to come!
</ul>
</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
<p id="default">
Have an Analytic you want to contribute? Interested in making your mark?
Contribute! Please reach out to Stephen to get added to the working team.
</p>
</span>
</body>
</html>